A Post-Mortem Evaluation of an IT project A Case Study of a Process Enhancement IT-Project In a Maintenance, Repair and Overhaul Company

The present work represents a post mortem evaluation of an SAP-IT project. It focuses on critical success factors (CSF) in order to establish an appropriate guideline for the project evaluation. A review of contemporary project management literature identifies general project CSF and SAP-project specific CSF and provides a brief theoretical overview of the purpose of project reviews and points out difficulties regarding performance measurements. The review of the project is a qualitative evaluation based on selected CSF. Project evaluation is an ongoing, multidimensional process and can be used to measure success and to learn from previous experience. The CSF used to measure the project success, need to be well defined and constantly checked as they can change over time. For project success a good communication with all stakeholders involved is fundamental. Constant review becomes necessary in today’s complex project world in order to engage in total quality management and continuous improvement.


BACKGROUND OF THE CASE STUDY PROJECT
In 2004 a one year IT project was conscribed at a business unit of German MRO Company in order to improve the overall material process.The project had a budget of about €1Mio ( £0.68 Mio) and aimed for the conversion of previous IT solutions into the SAP-standard system to simplify, enhance and condense the material ordering process.The project not only introduced a new IT system but also meant organizational change.The main stakeholders of the project were the Engineering Department, Disposition, Purchasing, Production and Warehousing.

AIM AND STRUCTURE OF THE STUDY
The aim of the present study is to conduct a post mortem analysis and final project evaluation of the case project.A focus is given on the determinants of project success and failure (Critical Success Factors -CSF).Firstly a literature review is given of the contemporary project management literature about CSF and project evaluation.This includes a project definition and the project lifecycle theory.An appropriate framework is presented in order to understand CSF, to enable a qualitative evaluation of the project and as an effort to narrow down the complexity of the topic.
Secondly the framework for evaluating project success is applied to the project.This analysis consists of a personal review and feedback collected from previous colleagues and project stakeholders.
Finally, some of the learning outcomes of the post mortem evaluation are summarized and recommendations for future project work are derived.
The author was involved in the case study project from the start up phase to the close phase and used the newly introduced IT-system for approximately one year after the project closure.The analysis is qualitative and reflects the personal opinion.

PROJECTS AND THEIR CRITICAL SUCCESS FACTORS (CSF)
Traditionally a project is defined as "an undertaking to achieve a defined objective, and goes on to state that 'generally all projects evolve through a similar "lifecycle" sequence during which there should be recognized start and finish points" [Turner and Cochrane, 1993].
This and similar definitions are based on the assumption that the project objectives are clearly defined.The project success could then be measured against the agreed objectives which are usually centered on the iron triangle of project management: Quality, Cost and Time (see Figure 5-1).
"Prior to the 1980s it was common to focus exclusively on project performance, which was defined narrowly as meeting cost and time objectives and adhering to a product specification" [Bryde, 2003].But project success is multidimensional [Bryde, 2003, p.229] and "in the late 1980s, after the introduction of TQM [Total Quality Management], a project was considered to be a success by not only meeting the internal performance measures…."[Tukel and Rom, 2001, p.400]."For example, in Wateridge's [1995] study of the impact of success criteria on a number of information technology (IT) projects, he concludes that the customer and other stakeholders, such as users, will define what they mean by quality" [Bryde, 2003, p.230].
A very general framework to analyze performance, and therewith linking to key dimensions of project success, is the EFQM model [Bryde, 2003, p.232] which originates in quality management concepts.The model is visualized in Figure 5-2 and it is used by organizations to evaluate quality aspects of processes, leadership and for project reviews.
Project management embraces various schools of thought; thus many different ways of how to approach a project review and of how to evaluate project success can be found in the academic literature.An overview of the development of the CSF-research is given in Table 5-6.This present research focuses on approaches that establish a CSF list and are appropriate to analyze the project performance of the case study.A different angle for instance on project success (strategic approach) was investigated by Jugdev and Müller [2005], suggesting that a successful project must add product, service and strategic value to the company.Some literature also distinguishes clearly between project performance and project manager performance.This present work recognizes this approach but does not make this differentiation in order to simplify the analysis and to provide a holistic view of project success factors.Rather, the work utilizes leadership performance as one CSF to evaluate project success.Again, "methods and techniques for evaluating projects have appeared in the literature for at least 40 years in hundreds of articles.Approaches tend to be either quantitative or qualitative, ranging from rigorous operations research to social-science-based interactive techniques (Henriksen and Traynor, 1999;Danila, 1989;Schmidt and Freeland, 1993).[…] "It is a tremendous task to evaluate the value of a project in detail."[…] "It should be noted that the first step in implementing project evaluation is to determine the factors against the projects" [Liang, 2003, p.446].Table 5-1, Table 5-2 and   Table 5-3 list and further describe CSF.Some of the criteria which can be evaluated by the author's observations will then be used to review the present case study project (see chapter 0).The tables of CSF are overlapping, clarify the complexity of this topic and show links to the quality model: EFQM (see Figure 5-2).
The initial definition of a project (see page 58) also includes the project life cycle, which is visualized in Figure 5-3."Previous research results indicate that the relative importance of several of the critical factors changes significantly, based on life-cycle stages (Pinto & Prescott, 1988)" [Hyväri, 2006].This is also indicated in the tables (Table 5-1 or Table 5-2) as certain CSF (e.g.Project Schedule and Plan) belong to certain project stages.Gardiner [2005, p.297] emphasizes on the wide variety and types of projects."Consequently, any list of success or failure factors should be used as a guiding principle only and modified according to the nature and context of each project …." Therefore and in order to evaluate the case -project with adequate and specific variables, this succeeding chapter 0 includes some CSF for SAP projects.

CSF FOR SAP PROJECTS
The study of Vidyaranya [2005] analyzed 44 published articles of companies that implemented the SAP system.He "identifies six common factors that are indicative of successful or non-successful SAP implementations.It has been found that the lack of appropriate culture and organizational (internal) readiness as the most important factor contributing to failure of SAP implementations in 15 companies."A summary of the six CSF for SAP Implementations is compiled in Table 5-3.

THE PURPOSE OF PROJECT REVIEW AND EVALUATION
"The processes of review and evaluation are applied at different stages throughout a project …" [Gardiner, 2005, p.296].Types of project evaluation are [Cicmil, 2007, b]: (1) Pre-project evaluation (2) On-going project evaluation (3) Project completion evaluation (4) Post-project evaluation (5) Post-mortem evaluation "The end of a project marks the last major milestone and provides an important opportunity to capture lessons learned during the project…."This is the motivation for the present work."It is also an opportunity to revisit the project's critical success factors" [Gardiner, 2005, p.296].The idea of the review also includes the continuously improvement approach."Evaluation is an objective, periodic stock taking to determine the status of a project in relation to its specific goals, taking into account project success criteria and recommendations for improvements of ongoing and future projects" [Cicmil, 2007, b].
Although one can find distinctions between project control and evaluation, Figure 5-4 visualizes the project evaluation/control cycle (also compare to: Figure 5-5 and Figure 5-6).Project control and evaluation are irreplaceable for project success as the planning can always only be a "good guess".
The next part identifies some general difficulties with (performance) measurements, which have to be taken into account for project evaluation.The following chapter then (chapter 0) analyses and evaluates the case study project against the CSF from the previous literature review.Conclusions are then drawn from this post mortem evaluation containing learning outcomes and future managerial implications.

THE TROUBLE WITH (PERFORMANCE) MEASUREMENTS
This subchapter refers to the article of Hammer [2007] "The 7 Deadly Sins of Performance Measurement" and provides fundamental criteria for effective and objective measurements.The findings are useful to identify suitable CSF and to evaluate project success appropriately.
According to Hammer [2007], the seven most common measurements mistakes are: Vanity: "measures that will inevitably make the organization, its people and especially its managers look good" Provincialism: "measuring narrowly in organizational boundaries" Narcissism: "measuring from own point of view rather from customer/stakeholder point of view" Laziness: "assuming one knows what is important to measure without giving it adequate thought or effort" Pettiness: "measure only a small component of what matters" Inanity: "Many companies seem to implement metrics without giving any thought to the consequences of these metrics on human behavior and ultimately on enterprise performance." Frivolity: "not being serious about measurements, passing the blame to others" Summarizing one can say: Identifying the right CSF and measuring/evaluating them is associated with great effort but inevitable for project success.Creating a measurement friendly culture and creating the right metrics is another challenge for a project manager.

Evaluation of the Case Study Project
The three project goals of the case study project were: (1) Optimization of the overall process of materials allocation (2) Conversion of all past IT solutions to SAP-Standard by February 2005 (3) Continuous illustration of the materials allocation process in SAP -from the parts list to the supply stock storage These three project goals were achieved within the time frame, the budget and with an appropriate quality.However, to evaluate the overall success of the project some critical success factors have to be reviewed (see Table 5-4).
Table 5-4 represents a personal, qualitative review of the case study project.For an evaluation ten appropriate CSF identified previously in the literature were selected.Then a descriptive evaluation for each individual CSF is given and the performance-level of each factor is evaluated on a scale from one to ten, with ten meaning that the CSF was fulfilled 100%.
Finally an overall project success evaluation is provided and the personal opinion is compared to some feedback given from colleagues that are currently working with the IT-system.
Overall one can say that the case study project was successful.The project objectives were met within the Iron Triangle (cost, quality, time) and most of the CSF reviewed (see Table 5-4) have been considered during the project execution.However an average "performance -score" of 5.23 out of possible 10 reveals that not all the potential of the project was exploited successfully.The project had a difficult delayed start and was executed within a difficult environment of uncertainty, missing trust, unclear requirements and low commitment of the Engineering Department (part of the stakeholders).
Due to the time pressure adequate testing and extensive training of the SAP system was minimized which explains today's difficulties in using the system among the stakeholders (see feedback Table 5-5).Communications across the various departments and different stakeholders is still far from optimal, although the new SAP layout simplified the processes (see feedback Table 5-5).
The change of the external environment and a missing continuously improvement program, including teaching and system adaptation, causes frustration and a blame culture among the users (see feedback Table 5-5).
Compared to the previous "IT system maze" and the complex material ordering process of this MRO-Company, the new SAP system certainly increased the performance and quality of the internal processes and material traceability, which is the biggest argument for the project success.
On the other hand a majority of the stakeholders are still either not familiar with the system or annoyed by its limitations and inflexibility which indicates that not all CSF were met or regarded with the same importance.

Conclusion
Projects are used to manage all different kinds of change.Critical Success Factors can be used as a framework to measure project success and are a very useful tool for project managers to effectively manage projects.CSF change over time, can require high skills and expertise and furthermore depend on the type of project.The project evaluation is a process going through all phases of the project life cycle (project control) and can also be used in a post mortem evaluation to learn from the previous experience and to engage in a continuously improvement process (TQM).
Important lessons learned by the literature review and future implications could be summarized as follows (also see [Jugdev and Müller 2005, p.29]: (1) Define a certain CSF framework to be able to measure the project success throughout the various phases of the project cycle (2) Identify key project stakeholders and allocate them to a certain category of the CSF (3) Project success is multidimensional and CSF need to include efficiency and effectiveness measurements regarding all project phases and all stakeholder (4) CSF may change over time between initial phase and closure phase (5) A good relationship and good communication with all stakeholders, including teamwork, is essential for the project success The SCOPE post mortem evaluation clarified the importance to break down a project in certain aspects (CSF) in order to evaluate the overall project success.Achieving time, cost and quality objectives does not necessarily mean that all stakeholders are satisfied with the project.Also, a project that is called "successful" does not coevally mean that all requirements are met.
The complexity of today's projects and the constantly changing environment create a situation in which it is fundamental to have a set of critical factors (clearly defined goals, milestones, objectives, CSF) against which the project success can be measured.
Constant review and evaluation becomes necessary in order to establish a continuously improvement process (TQM) for the organization and for the project manager himself.Note 1."Project manager who employ transformational leadership and, more specifically, idealized influence, in conjunction with a relationship-oriented approach enjoy more project success …" [Prabhakar, 2005, p.57] Note 2. "Effective project manager leadership is an important success factor on projects.The capabilities of the people involved resolving extraordinary situations and unforeseen problems are an important key for project success…" [Prabhakar, 2005, p. 53].Technical Tasks Availability of the required technology and expertise to accomplish the specific technical action steps.

Client Acceptance
The act of 'selling' the final project to its ultimate intended users

Monitoring and Feedback
Timely provision of comprehensive control information at each phase in the implementation process

Communication
The provision of an appropriate network and necessary data to all key factors in the project implementation

Troubleshooting
The ability to handle unexpected crises and deviations from the plan

Additional four factors 'beyond the control of the project team'
Characteristics of the project leader Competence of the project leader (administrative, interpersonally and technically) and the amount of authority available to perform his/her duties

Power and Politics
The degree of political activity within the organization and perception of the project as furthering the self-interests of an organization's members

Environmental Events
The likelihood of external organizational factors impacting on the operations of the project team, either positively or negatively

Urgency
The perception of the importance of the project or the need to implement the project as soon as possible -How well are the requirements defined?-Clean up operations to implement "Vanilla-SAP" -Ability to maintain scope, related to the planning -2project team/management support/consultants -successful project team is cross-functional, -must be dedicated solely to the project -high-level executives have a strong commitment to the project -incentives for the team member and open internal communication channels -technical and people goals must be met -3internal readiness/training -People element and training aspect -Long run effects -Difficult to measure -Employees must be trained on system for day to day operations -Managers must know the implications of the system (enthusiasm) -reinforcement of a "team environment" is critical to the overall success -Readiness for change (cultural change by new system -control etc..) -4deal with organizational diversity -Individual branches, individual procedures in different departments -diversity can be obstacle to success -to re-engineer their processes and remove idiosyncrasies -both cultural and procedural -Before any company can be linked effectively to world-class supply chains, their internal processes must be world-class (Ptak, 2000).-Many large companies, Amoco and Chevron, for example, successfully reengineered their business and overcame the problem of organizational diversity.
-5planning/development/ budgeting -complex task -enormous potential costs -Major expenses incurred by companies that were unable to fully develop a comprehensive plan.-Planning should be closely identified with maintaining scope during an implementation.-Some companies in the midst of an implementation were forced to scuttle the operations and make quick fixes to their legacy systems.-Developmental delays can also lead to resource attrition, which in turns affects the learning curve and completes the vicious cycle by creating additional obstacles to obtaining cut-over.-Budget-plan: Only one-sixth of projects are completed on time and within budget (May, 1998).-6adequate testing -the key element of success for some companies, and a direct cause of failure for others => long run effects -risk: attitude of "just finish it", project-tiredness -testing and red flags ignored, pressure to meet timelines, top management support needed!The case project was closed within the agreed budget but the project also included unnecessary costs (e.g.personnel costs …) 6

Quality
The system fulfilled all the initial requirements; yet because of urgency, lack of testing and changing requirements (see below) the full system potential was not exploited and bugs were included.

Time
The planned project start was delayed by about two months because of "doubts" in the review board.However the project was finished and the IT system used on the agreed time.

Leadership
The project leader proved a great administrative, interpersonally and technical competence.He was committed, experienced, built up trust and had good communication skills.Also his situational management was excellent.However, due to power and politics in the company (see below) (and maybe confidence) he was missing authority and could not accomplish all the goals.7 Project Team/ Personnel The project team was cross functional as the stakeholders came from various departments.But, because of capacity reasons not all of them were solely committed to the project.
The team consisted of a lot of students which were highly motivated and committed but lacked of project experience and skills.Incentives for the team member were created and there was a good open communication within the team.However the team also included very low interested stakeholders which slowed down the project annoyed other members and increased the project costs.Technical knowledge and expertise was provided by consultants and programmers.

Organizational Factors
The company was involved in a lot of projects during the 1990s as part of the restructuring of the airline and various cost cutting programs.Overall it was proven that the company is ready for change.However this particular business unit, due to its pride and unsuccessful previous projects did not show much interest in and motivation for the project.The IT implementation was more complicated due to retracted, obsolescent and very bureaucratic processes.

External Environment
The requirements for the IT system changed during and after the project.A quick adaptation was impossible.The biggest change was that the company changed the way of production.The material ordering system of SAP was not designed/prepared for this change in the "external environment" 2 Client Acceptance The client acceptance was/is very different: Engineering: Low acceptance because of the dislike of a further IT system and the fear of being controllable Management: High acceptance but little interest in learning the system themselves -most of the management (lower and higher) does not know how the new system works Disposition: This position was newly created as a link between Engineering and Purchasing -highest acceptance and key position of new system Purchasing: High acceptance as the new SAP system simplified and structured their work compared to the previous processes Power and Politics The company is coined by a lot of politics and bureaucracy.It is very difficult to implement change and to accomplish goals as a new leader without much power.Furthermore the project included people only working for their personnel aims. 3

Urgency
The project was perceived as highly important for the management and there was enormous pressure to finish the project on time.Because of the delayed start by the review board the project process had to be accelerated and for example testing and "change management" suffered in the end.The project leader provided a risk analysis for finishing the project on time and the decision was made to stay in schedule and accept bugs and teething problems.

SAP Functionality/ Requirements/ Testing
The system requirements were not clearly defined by the clients.A lot of meetings were necessary to gather the information.The specification book was written by a student more as a summary than as a guideline for the programmers and consultants.Training and a training book was provided for the stakeholders in a fairly good quality.However the system testing actually started once the finished system was switched on.

Communication
The communication within the project team was very good (Teambuilding Events!).The communication outside the team suffered a little bit from missing trust, respect and commitment towards the project team and the new members.
(This changed when e.g.students proved their competencies) 7

Table 5 -
5: Some Interview Answers from Previous Research