Evaluation of the Success of Six Sigma Projects by Data Envelopment Analysis

Six sigma provides a methodology for problem solving by using project management tools systematically. The primary aim of six sigma applications is minimizing defects. Minimizing defects will ensure the outputs and customers’ satisfaction increase and the costs decrease. However, the number of the firms that state that they couldn’t get the results from six sigma programs as they expected is not few. The main reason for failure of six sigma programs is not to balance the costs with gains of six sigma projects. So, evaluating the performance of six sigma projects is an important issue for the firms that apply six sigma programs. In this paper, data envelopment analysis was used to evaluate the performance of six sigma projects which is stated as critical success factor for six sigma programs. Data envelopment analysis which is based on linear programming helps to determine the efficient and inefficient decision making units by comparing the relative efficiencies of decision making units. Recently, there are some papers in the literature which apply data envelopment analysis for evaluating the performance of R&D and software projects. There are also some papers which discuss the application of data envelopment analysis in the selection of six sigma projects. However evaluating the performance of six sigma projects with data envelopment analysis was not discussed comprehensively in the literature. This paper is original by showing that data envelopment analysis which can be used in evaluating the performance of organizations can also be used for evaluating the performance of six sigma projects. The application of data envelopment analysis for evaluating the performance of six sigma projects was shown by a case study.


Introduction
Inputs are transformed into outputs in the processes of the organizations.In this transformation, the inputs such as labor, material, machine and energy are used and products/services are acquired at the end of the transformation.The value of the output must be more than the value of the inputs used.The managers of the organizations want to evaluate their efficiency according to specific performance measures.Performance measurement helps organizations to see which improvements they should do in addition to providing information about the current state of the processes.
Companies bear important costs with the application of six sigma programs and expect high revenues after six sigma programs.The firms that want to enhance high revenues with six sigma programs should achieve successful six sigma projects.The success of six sigma programs are strongly related with the six sigma projects.However there may be many six sigma projects in firms which didn't provide the expected gains.The six sigma programs can be stated as unsuccessful after the six sigma projects that did not provide the expected benefits.For this reason it is important for the firms to evaluate the performance of the six sigma projects.
Data envelopment analysis (DEA) is an approach used in organizations to evaluate their performance according to others.In the literature DEA which is a nonparametric techniques was used to evaluate the performance of many different decision making units such as hospitals, education institutes, municipalities and countries.DEA can also be used to evaluate the performance of six sigma projects.After six sigma projects many benefits such as an increase at customer satisfaction and revenues and a decrease at the costs are expected.For enhancing the expected outputs, labor hour and financial resources are consumed.By evaluating the performance of six sigma projects by DEA it can be possible to determine efficient and inefficient six sigma projects and the causes for inefficiency can be stated.

Six Sigma
Six sigma programs provide to reduce the variation in the processes and six sigma means 3,4 defects are produced for million opportunities in the processes.Six sigma presents a problem solving methodology in order to eliminate root causes of the defects.By this way it can be possible to reduce the defects and costs associated with the defects.Six sigma should be seen as a management philosophy at the same time.Six sigma is a customer focused approach that emphasizes that defects are factors that reduce the customer satisfaction and increase the costs (Brue and Howes, 2006).Six sigma helps the organization to gain more money by increasing the customer value and efficiency.Six sigma focuses on improving the quality by helping the organizations to produce their products and services faster , better and lower costs (Pyzdek 2003).Many organizations apply six sigma for the financial gains to be enhanced.Six sigma programs have ensured achievements in four areas in the firms; financial gains, benefits from the viewpoint of customers, benefits from the viewpoint of personnel, and effects of quality.It is estimated that six sigma applications can provide gains three or four times more than the cost of applications (Brue and Howes, 2006;11).The firms which are at 3 or 4 sigma level spend 25-40 percent of their revenues for solving the problems.Besides the firms at 6 sigma level spend less than 5% of their revenues for solving the problems (Pyzdek, 2003).Banuelas and Antony (2002) stated critical success factors for six sigma applications as, management involvement and commitment, cultural change, organization infrastructure, communication training, linking six sigma to business strategy, linking six sigma to customers, linking six sigma to human resource, linking six sigma to suppliers, understanding tools and techniques within six sigma, project management skills, project prioritization and selection (Banuelas and Antony,2002).According to John and Swicher (2003) the continuous and visible commitment of management, continuous education through organization, understanding the customers' expectations, selection of projects leaders carefully and selection of projects which are strategically important are the key factors for successful six sigma applications (Johnson and Swisher,2003).Based on various literature review and discussions with six sigma leaders, Kwak and Anbari (2006) identified key elements of six sigma applications as management involvement and organizational commitment, project selection, management and control skills, encouraging and accepting cultural change and continuous education and training (Kwak and Anbari,2006).In the literature most of the papers that discuss the critical factors for six sigma applications emphasize the importance of six sigma projects and the process of six sigma projects' selection (Banuelas and Antony, 2002).

Six Sigma Projects
The success of six sigma applications is strongly related with the six sigma projects that provide financial gains to the firm and increase the customers' satisfaction.Six sigma projects provide significant revenues to the firms but at the same time significant time and financial resources are needed for six sigma projects.So, the firms expect significant gains over against labor time and financial resources that were consumed.However there are many firms that stated that the six sigma programs had been very expensive for their firms and they couldn't have achieved the expected gains In spite of the popularity of six sigma applications in manufacturing and service firms, there is also a concern about unsuccessful six sigma applications.According to a research in aerospace industry less than 50 percent of the firms, attended to the research, were satisfied with six sigma programs (Zimmerman and Weiss, 2005).According to Angel and Pritchard (2008) 60% of the six sigma applications could not achieve the expected benefits from six sigma applications (Chakravorty, 2009).
The main reasons of many six sigma programs and projects are related with the goals not defined clearly and selection wrong six sigma projects (Zimmerman and Weiss, 2005).Selection and prioritization inappropriate six sigma projects may cause unsuccessful six sigma applications because of not linking the expected benefits from the projects with the strategic goals (Gijo and Rao,2005).According to the research done by Raisinghani et al (2005) the firms were asked how they select six sigma projects.92 percent of the firms stated linking with strategic goals, 86 percent of the firms stated linking with expected financial gains and 76 percent of the firms stated linking with customer satisfaction (Raisinghani et al., 2005).Banueals et al (2006) examined how six sigma projects are selected, prioritized and evaluated in UK.Six sigma projects often begin with the determination of customer requirements and it is essential to set project goals based on reducing the gap between the company's deliverables such as quality, delivery time, reliability and customer expectations.According to the research, about 60 percent of the companies identify potential projects from their customers.The critical factors in the selection of six sigma projects were determined as customer satisfaction, financial gains, linking with organization strategy and top management commitment (Banuelas et al.,2006).
Selection of six sigma projects is a very important stage.If this stage isn't done as required the gains from the six sigma projects won't be achieved besides consuming time and resources.In the literature the importance of the stage of selection of six sigma projects has been emphasized for the success of six sigma applications and selection and prioritization of six sigma projects has been determined as a critical factor for success of six sigma applications in many articles.

Selection and Evaluation Process of Six Sigma Projects
As management decisions of all projects the management decision of six sigma projects involves four resources; time, resources, cost and performance.The start and finish times and the times for the activities of the projects must be known to meet the due date of the projects.The requirement of the labor and equipment for the project and assignment of these requirements to the activities must be determined.The budget and cost should be tracked because all the projects have got a limited budget and the cost occur according the resources consumed.Performance should be evaluated for determining how the results of the project met the customers requirements (Evans and Lindsay, 2005).
Six sigma projects should have at least one of the goals, reducing the variability, reducing the defects, increasing customer satisfaction, decreasing the costs, increasing cycle time, increasing flow output, decreasing lead time, optimization of process performance and optimization of supply chain (Brue and Howes,2006).Six sigma researches the ways for improving the external performance that affects the customers directly such as improving product designs and fast deliver and internal performance such as increasing productivity and decreasing the defects (Evans and Lindsay, 2005).The firms focus to evaluate the financial and nonfinancial benefits in the selection of six sigma projects The effects of a six sigma project can be evaluated in three groups.Most of the projects are selected according to positive effect for the customers provided.Six sigma projects allow the firms to increase the revenues and efficiency.Improving the value delivered to customers increases the revenues and decreasing the cycle time, eliminating waste in the processes and minimizing the cost of quality improves the efficiency of the processes.So, gains in the revenues and improvements in efficiency can also be important in the selection of six sigma projects in addition to the value delivered to customers.Improving the work satisfaction of the workers, environmental laws, legal issues can also be considered in the selection of six sigma projects (Pyzdek, 2003).
The measures that will be used in the evaluation of the success of six sigma approaches should be determined.Six sigma champions should determine and deploy the processes that will ensure the maximization of the investment returns of six sigma projects.Financial gains may be the most important factor in the measurement of six sigma results in many organizations.Generally the measures for evaluating the six sigma programs are preventing cost drivers, decreasing costs, increasing capacity, increasing revenues and risk management (DeRuntz and Meier, 2010)

Data Envelopment Analysis
Data Envelopment analysis (DEA) is a methodology based on linear programming for measuring and evaluating the relative performance of organizations that consume identical inputs for producing identical outputs.The roots of DEA principles were based on the research done by Farrel in 1957 and the studies in the literature about DEA was started with the article written by Charnes -Cooper -Rhodes in 1978.The CCR model developed by Charnes-Cooper-Rhodes was the first mathematical model of DEA.
Data envelopment analysis is an approach for evaluating the performance of a set of peers called Decision Making Units (DMUs) which convert multiple inputs into multiple outputs.(Cooper et al, 2004;1).There are many articles in the literature about the application of DEA in evaluating the performance of hospitals, education institutes, universities, manufacturing companies and service companies, municipalities, cities and countries.
The performance of DMUs is assessed in DEA using the concept of efficiency which is the ratio of total outputs to total inputs.Efficiencies estimated using DEA are relative to the best performing DMU (Ramanathan, 2003).DEA calculates the amount and type of cost and resource savings that can be achieved by making each inefficient unit as efficient as the most efficient units (Sherman and Zhu, 2006).The best performing DMU is assigned an efficiency score of unity or 100 percent and the performance of other DMUs vary between 0 and 100 percent relative to this best performance (Ramanathan, 2003).
DEA methodology determines the best applications among the DMUs and expresses production frontier by solving mathematical program.The production frontier data set determined by the model envelops all the DMUs and allows to compare all the DMUs for their performance.The efficiency of a unit is evaluated according to the distance to the unit it is compared.The efficient frontier expresses the relationship between the inputs consumed and the outputs produced.It shows the maximum quantity of outputs that can be produced from a specific combination of inputs.At the same time it expresses the minimum quantity of inputs that must be used to achieve a specific output level.
The efficient frontier is determined by the most efficient DMUs under study.A specific DMU is efficient if there is no other DMU or combinations of other DMUs which can produce at least the same amounts of outputs with less of some resource input and no more of any other resource.A DMU is said to be inefficient if another DMU or combination of other DMUs can produce at least the same amounts of all outputs with less of some resource input and no more of any other resource (Weber, 1996).
DEA is a methodology directed to frontiers rather than central tendencies.DEA proves particularly adept to uncovering relationships that remain hidden from other methodologies.It doesn't require formulated assumptions and variations with various types of models such as linear and non linear regression models (Cooper et al., 2004).DEA is a nonparametric method.Solution obtained using parametric methods are very sensitive to a prior weights assigned to the inputs and outputs.DEA, as a nonparametric method, does not assume a prior weights for inputs and outputs (Kumar et al., 2007).A typical statistical approach is characterized a central tendency approach and it evaluates the organizations to average performance.DEA compares each organization wit the best performer.
The formulation can be either input -oriented or output-oriented.For the input oriented case the linear programming formulation checks whether a hypothetical DMU exists whose outputs are as great as the DMU under consideration by consuming lesser input.For the output oriented model, the linear programming formulation checks whether it is possible to create a hypothetical DMU which uses the same quantities of input resources and produces more outputs than the output quantities produced by the DMU under consideration (Kumar et al., 2007).
DEA has two basic models; constant returns to scale (CRS) and variable returns to scale (VRS) models.CRS model developed by Charnes et al. (1978) assumes constant return to scale.Banker et al. (1991) suggested VRS model that assumes variable returns to scale.

CCR Model
Constant returns to scale means that the organizations are able to linearly scale the inputs and outputs without increasing or decreasing efficiency.In this paper input oriented CCR model is formulated according to the application of the paper

Input Oriented CCR Model
Decision making units express the units that will be evaluated according to ability of transformation of inputs to outputs.DMU j states the decision making unit number j.It is presumed that there are n DMUs and all the DMUs use m different inputs which quantities are different to produce t different outputs.The DMU j uses the amount x ij of input i and produce the amount y rj of output r.It is assumed that x ij ≥0 and y rj ≥0 and every DMU has at least one positive input and one positive output value .The decision making unit which is evaluated shown by "0" and the other DMUs shown by "j".The ratio of outputs to inputs is used to measure the relative efficiency of the DMU j = DMU 0 to be relative to the ratios of all of the j=1,2,……….nDMU j (Cooper et al.,2004).
For a specific decision making unit, the mathematical programming problem can be stated as, .n u r , v i ≥0 for all i and r j= number of decision making unit being compared in the DEA analysis DMU j =the decision making unit number j. h 0 = efficiency rating of the decision making unit being evaluated by DEA(efficiency rating of DMU 0 ) y rj = amount of output r used by decision making unit j x ij = amount of input i used by decision making unit j i= number of inputs used by the DMUs r = number of outputs generated by the DMUs u r = coefficient or weight assigned by DEA to output r v i = coefficient or weight assigned by DEA to input i Linear programming is the underlying methodology that makes DEA particularly powerful compared with alternative productivity management tools.The linear programming technique is used to find the set of coefficients (u's and v's) that will give the highest possible efficiency ratio of outputs to inputs for the decision making unit being evaluated (Sherma and Zhu,2006).
Because of the difficulties of solving fractional mathematical programming model is transformed to linear programming model The dual model can be given as follows;  are the dual variables.The variable θ, is the technical efficiency score which is calculated and  i s =  r s =0 for all i and r. θ=1 ensures that DMU 0 is located on the production frontier and if θ≠1 then DMU 0 isn't located on the production frontier and an adjustment can be made in order to move inefficient DMU 0 to efficient frontier.The following input/output adjustments (improvement targets) would render it efficient relative to other DMUs; The CCR model (constant returns to scale) assumes that the DMU under consideration is operating at the optimal scale.Banker, Charnes and Cooper modified the CCR models to account for variable returns to scale.This model is known as the Banker-Charnes-Cooper (BCC) model.The BCC model, on the other hand, allows DMUs to be classified as efficient even if they are not operating at the optimal scale size (Sherma and Zhu,2006).An additional convexity constraint can be added to the equation to yield a measure of the pure technical efficiency if the constant return to scale assumption does not apply (Liu et al., 2000).
The following convexity constraint is added to the envelopment formulation:

Evaluating the Success of Projects with DEA
DEA can be used to evaluate the relative performance of projects by combining the multiple inputs and outputs that affect the performance of projects (Xu and Yeh, 2011).DEA can overcome the paradigm of project uniqueness by providing organizations with a method of accounting for differences in project input characteristics when measuring performance across projects (Farris et al., 2006).
DEA creates comparisons groups by utilizing projects with similar input characteristics and comparing the performance of each project to its most similar peers.DEA doesn't require that all projects combine inputs exactly the same way to produce outputs, thus allowing the flexibility necessary for the analysis of a complex task (Farris et al., 2006).
According to literature, there are articles that use DEA to evaluate the performance of design and software projects besides the changes of the design activities among the projects.There are few articles that discuss the use of DEA in the projects.Busby and Williamson (2000) used DEA in measuring the performance of engineering design activities.Paradi et al. (2002) stated DMUs as engineering teams and used DEA in the evaluation of technical and allocative efficiency of engineering teams.Farris et al. (2006) used DEA as a tool in the evaluation of the projects in an engineering department.Linton et al. (2007) determined that DEA can be used in evaluation, prioritization of the projects in research and development department.Eilat et al. (2008) suggested a combination of balanced scorecard and DEA in the evaluation of efficiency of the projects in industrial research department.Xu and Yeh (2011), Shirouyehzad and Dabestani (2011), Mahamud et al.(2011) used DEA in the evaluation of projects efficiency.Chinubhai (2011) suggested to use DEA in the evaluation of software development projects efficiency.Cao and Hoffman (2011) used DEA in determining efficient and inefficient engineering projects, in presenting the reasons for inefficient projects and in analyzing the factors that hinder efficiency in engineering projects.
7. Evaluating Six Sigma Projects with DEA Kumar et al(2007) suggested to use DEA in the selection of six sigma projects and they showed the use of DEA in the six sigma projects' selection with a hypothetic example and took the outputs and inputs' expected values.Inputs were determined as project cost, project duration, number of black and green belts and outputs were determined as customer satisfaction, impact on business strategy, increase in sigma level, financial impact and increase in productivity.They determined inputs and outputs values for 20 hypothetical projects (Kumar et al., 2007).Farris et al. (2006) presume that engineering projects show constant returns to scale.Cao and Hoffman (2011) assumed that diseconomies of scale might exist for many input variables, for instance, increasing projects staffing beyond a certain level may yield diminishing returns in project duration, due to congestion.
In this paper, it was assumed that six sigma projects show constant returns to scale and input oriented CCR model was used.Input oriented model was used because technical efficiency wanted to be determined according to the specific set of inputs and as all the projects controlling the inputs is easier than controlling the outputs.For this study the DMUs are six sigma projects in the firm.DEA model was used to measure the relative efficiency of each six sigma project based on multiple criteria when compared to the most efficient six sigma project under investigation.One of the most important stage was determining the inputs and outputs for evaluating the relative efficiency of six sigma projects.The inputs were determined as labor hour for project and project cost (project cost states the investment costs and it was estimated that the projects evaluated in this example required investments) and outputs were determined as financial gains, increase in sigma quality and increase in customer satisfaction.The application of the model for measuring the performance of six sigma projects was shown by a case study in a firm.The inputs and outputs were determined for 5 six sigma projects in the firm.These 5 six sigma projects were selected from six sigma applications directly related in manufacturing processes and all six sigma projects that had been selected were similar according to content and the project duration.The input and output values for the case study were given in table 1.Data envelopment analysis online software (ttp://www.deaos.com/login.aspx?Return Url= %2fWelcome.aspx)hasbeen used for calculations.Table 2 shows the results of efficiency scores of 5 six sigma projects.According to the results, six sigma project 2, six sigma project 5 are determined as efficient six sigma projects and the other six sigma projects are inefficient.Peer group analysis has been performed to compare six sigma projects that are not efficient with projects that are efficient.The performance of the inefficient six sigma projects can be improved by comparing six sigma projects according to efficiency.Table 3 shows reference sets for inefficient six sigma projects Table 4 shows inefficient six sigma projects and their dual weights form input oriented CCR model.According to table 4, inefficient six sigma projects can be benchmarked according to six sigma projects that needed to be caught up to.According to table 3 to become efficient, six sigma project 1 must use a combination from both six sigma project 3 and six sigma project 5 (a virtual six sigma project) to become efficient.How much of SSP3 and SSP5 are calculated to achieve efficiency and reported next to each benchmark six sigma project.These are ‫ג‬ weights obtained from the dual versions of the linear program that is solved to estimate these values.SSP1 will attempt to become like SSP5 more than SSP3, SSP2 will attempt to become like SSP3 more than SSP5 and SSP4 will attempt to become like SSP3 more than SSP5 as observed from respective λ weights of efficient six sigma projects (Özcan,2008).Table 5 shows input and output efficient targets for input oriented model.For the input orientations, CCR model the efficiency score of inefficient six sigma projects can be increased and improved by reducing the inputs to a certain level.The excess inputs can be reduced and inputs can be minimized.The reduction of labor hour for SSP1, SSP2, SSP4 are 23,80%, 12% and 2,92% respectively.The reduction of project cost for SSP1, SSP2, SSP4 are 95%, 64,72% and 72,13% respectively.Additionally some of the outputs need to be increased in order to increase the performance of six sigma projects and efficiency scores.The output of financial gains for SSP4 should be increased 22%, the output of increase in six sigma level for SSP1, SSP2 should be increased 96,4% and 93,43% respectively.

Conclusion
There are many papers in the literature about the application of DEA in the evaluation of the performance of organizations.However the papers that discuss the application of DEA for the evaluation the performance of projects are very limited.In this paper DEA has been used for evaluating the performance of six sigma projects in the organizations.The organizations consume many resources for their six sigma programs and there are many companies that couldn't achieve the benefits and revenues that they expected from six sigma programs.The success of six sigma programs are strongly related with the six sigma projects.So, evaluating the performance of six sigma projects is very important for the organizations.In this paper, DEA has been suggested for evaluating the performance of six sigma projects in organizations.Input oriented CCR model was used and inputs and outputs were determined by a case study in a firm.According to case study the relative performances of the six sigma projects were evaluated and improvements were suggested for improving six sigma projects that were inefficient.One of the limitation of this case study is the number of DMUs according to total number of inputs and outputs.In the future with more DMUs the case study can be improved.The firms can use DEA for evaluating the performance of their six sigma projects and can determine the reasons for six sigma projects that are inefficient and can find ways to improve six sigma projects in their firms.

rs
are input slack and output slack respectively.Output slacks indicates how much shortages in the outputs and input slacks indicates how much surpluses in the inputs.DMU 0 is efficient if and only if , in the dual optimal solution θ * =1

Table 4 .
Inefficient projects and their dual weight values from input oriented CCR

Table 1 .
Inputs and outputs for 5 six sigma projects

Table 2 .
Input oriented CCR efficiency scores for 5 six sigma projects

Table 3 .
Reference sets for inefficient six sigma projects

Table 5 .
Input and output efficient targets for input oriented model