A Multi-Criteria Model to Evaluate Public Services Contracts

A major trend to improve Public Administration has been the increase of contracting out services hoping to achieve better levels of performance. However, the effectiveness and efficiency of this approach implies the application of appropriate models to evaluate such performance This is why a multi-criteria model was developed by the authors to evaluate and to improve the performance of public services contracts focusing on four key dimensions: the process of contract formation, the contract costs, the benefits achieved by the contract execution and its impacts. The proposed model provides a stable, consistent and integrated framework allowing not just the evaluation of contracts but also the identification of the priorities for improvement. A successful application to the evaluation of Home Respiratory Services is also presented.


Introduction
Modern Economy is built on exchanges of information and values based on digital networks and therefore its development requires a myriad of diversified contracts to regulate and control such contracts. The value of contracted services is growing fast for most organizations, namely in the public sector as they are contracting out a wider spectrum of specialized services (Geys & Sørensen, 2016;Kuek et al., 2015;Vincent-Jones, 2006). This explains why the value of public procurement in the European Union exceeds more than 13% of the GDP (excluding utilities and some concessions) in 2017 (Directorate-General GROW, 2019) showing an increase of over 6% since 2014.
The new Directives on public procurement approved by the European Union in 2014 (European Parliament and the Council of the European Union, 2014) have included several innovations to modernize public contracts, namely new rules about the evaluation of contract performance, which is a key condition to assess the merit of each contract and to elicit the development of guidelines. The improvement of the procedures to form and execute public contracts, namely public services contracts, and the achievement of high value for money implies a continuous learning framework to promote self-auditing about the adopted procedures and the evaluation of the results of the contract execution.
The proposed model is applied to a real case concerning the evaluation of an executed contract of public services concerning Home Respiratory Services in Portugal.

Literature Review
The two types of decision mentioned before fall into the areas of classification and sorting which have been studied by several authors of Multi-Criteria Decision Analysis (MCDA) (Arnott & Pervan, 2014;Greco et al., 2016;Zopounidis & Doumpos, 2002). However, according to these authors, the areas of applications of such a wide range of models include "medicine, pattern recognition, human resources management, production systems management and technical diagnosis, marketing, environmental and energy management, ecology and financial management and economics" but the field of contracting is absent. Similar conclusions can be obtained from the extensive review of applications (Greco et al., 2016) including a list of applications areas not covering contracts and (Velasquez & Hester, 2013) present a long list of key applications without mentioning the concept of contract.
The main areas of development of Decision Supporting Systems (DSS) were reviewed by Arnott and Pervan (2014) concluding that the areas of Knowledge Management, Business Intelligence and Business Analytics are getting further attention but, unfortunately, issues on contracting are ignored most often.
However, studies were published on the application of MCDA and DSS to other decision problems connected to the process of contracting, namely on multi-criteria models to select candidates (Cheng et al., 2020;Hasnain et al., 2018;Montano et al., 2008;Tavares & Arruda, 2022), to select contracts (Pastor-Ferrando et al., 2010), as well as on bid evaluation (Asadabadi & Zwikael, 2020;Jimé nez et al., 2007), on multi attribute auctions in contracting (Chen et al., 2011;Yang et al., 2014), on past performance evaluation  and on negotiation through e-sourcing (Reyes-Moro et al., 2003) but not on ex-post contract evaluation. Only a few authors have published monographic or descriptive studies rather than proposing structured instruments meeting the objectives of this research (Eriksson et al., 2017;Greve & Coelho, 2017;Rendon, 2015).
Furthermore, the execution of public contracts should also consider specific principles stemming from the public nature of the allocated resources according to the rules of the EU Treaty (European Union, 2012) and of the mentioned Directives:  Sustainability and Innovation: These principles should concern not just the ecological and environmental dimension but also the social and labour perspectives as it is well presented by the Agenda EU 2020 (European Commission, 2010) as well as the contribution to innovative policies (Tavares, 2018(Tavares, , 2019.
 Equity: The public institutions should respect this principle by giving equal treatment to any competitor avoiding any privileges or unequal advantages and so the award of any contract should just depend on the merit of each tender.
 Transparency: The concept of transparency implies that any decision made by the public contracting authority should be made in terms of verifiable facts and that the reason for each decision will be made accessible to any competitor.
Unfortunately, no results can be quoted proposing MCDA models to evaluate the execution of public contracts coping with these objectives and principles and so the authors expect that this paper will contribute to fill up such gap using the general approach presented by Keeney (1992) and adopting the axiomatic assumptions of Multi Attribute Utility Theory (MAUT) (Dyer, 2016;Fishburn, 1970).

Research Question
The main research question addressed by the authors is: how can multi-criteria models be built to evaluate the performance of executed public contracts based on a stable and consistent structure allowing cross sector comparative assessments?

Main Hypotheses
Four main hypotheses were postulated for this research: 1. Multiple Stages: The concept of contract execution is formulated as a three stage process including the following components  Formation of the contract  Implementation of the contract object generating direct costs and benefits http://ibr.ccsenet.org Vol. 15, No. 3;2022  Impacts on sustainability and innovation 2. Multiple Objectives: The concept of contract performance should be defined in terms of four major objectives ( Figure 1):  Compliance with the principles of public contract formation, namely equality and transparency  Maximization of the expected benefits  Minimization of contract cost  Maximization of the contract external impacts to sustainability and innovation 3. Criteria Tree (Structure and Descriptors): Each objective should be described by a criterion which can be branched down into several sub-criteria and so each of the most elementary criteria should be evaluated in terms of the estimated score for an appropriate descriptor. The scoring for each descriptor is done using the same arbitrary cardinal scale (0 to 10) corresponding 0 (or 10) to the lowest (or the highest) level of preference.
4. Multi-Attribute Utility Theory (MAUT): The integration of the scores obtained for several criteria or sub-criteria is done assuming the linear compensatory hypothesis of MAUT (Dyer, 2016;Fishburn, 1970) which can be expressed by: for each evaluated contract i with i=1,…, N being N the number of evaluated contracts, with j=1,…M, being M the set of criteria, V(i,j) the score obtained for (i,j), V(i) the obtained integrated score for i and w(j) the weight assigned to the criterion j (0 ≤ (j) ≤ e ∑ (j) )

Structure
The development of the proposed model follows the approach proposed by Keeney (1992) recommending that the criteria will stem from the values and objectives to be pursued and representing a relevant point of view of the decision maker (Roy, 2016).
Therefore, the four main pillars objectives already presented imply the formulation of four main criteria in terms of: 1. The quality of the procurement procedure including the stage of contract formation and the conditions defined to initiate the contract execution; 2. The benefits generated by the contract execution in terms of the quality of the services provided; 3. The cost of the contract execution.

The contract contribution to sustainability and innovation;
In the following sections each indicator will be defined, and its descriptor specified.

The Formation of the Contract
The formation of any public contract requires the compliance with a set of very strict guidelines and legal rules http://ibr.ccsenet.org International Vol. 15, No. 3;2022 due to its public nature which are much more demanding and complex than those respected by private contracts, and this is particularly true in European Union Member States with a legal system based on public law instead of civil law as it happens in Latin cultures. The proposed criteria for the evaluation of contract formation are: 1. The level of accessibility for participation of potential candidates; 2. The level of transparency of the award decision; 3. The possibility of negotiating prices for specific levels of quality; 4. The level of litigation after contract award; 5. The level of contractual stability described by the duration of the contract execution; 6. The duration of the process of contract formation; 7. The level of clarity and consistency of the procedure documents specifying how the contract should be executed after being awarded.

The Conditions for the Contract Execution
The proposed sub-criteria are: 1. The clarity and consistency of the manual specifying how the services should be provided by the contractors.
2. The degree of digitalization and paperless in the processes connecting the contractors with the public contracting authority for the tasks of ordering, control and payment required by the contract execution.
The corresponding indicators are represented by: IE(1) and IE(2)

The Contracts Benefits: Quality of the Services Provided
The quality of the service contracted is a key dimension of the benefits as it corresponds to the "raison d´être" of the contract and it should be represented by the perception of such quality by the major beneficiaries and stakeholders. For instance, in the case of health services, an indicator should describe such perception by the patients (IQP) and the other concerning the quality perceived by their clinicians (IQC). In further developments, additional information should be collected to assess not just the quality but the impacts of the services on the quality of life of the final beneficiaries.

The Contract Cost
The cost of any public services contract is quite important as it measures the value of the public resources allocated to the contract. This criterion should be described in terms of the vector of unit prices, p(i), corresponding to each different type of service (i=1,…, N) being N the number of different types of services. Then, the total cost C is obtained by: being D(i) the amount of delivered services of type i.
The cost criterion (CB) should be formulated in terms of benchmarking with another equivalent contract selected for comparative purposes (R) and so it can be described by: being p*(i) the price vector for services type i, according to another contract adopted as benchmarking reference.
Therefore CB measures the savings as a fraction of the previous reference cost.

The Contribution to Sustainability and Innovation
The modern approach to public policies supporting innovation emphasize the so-called demand driven policies which are based on the introduction of innovative requirements into the demand specification and so public demand should also cope with this objective. Similarly, sustainability is a major public objective and so public contracting should also include this type of criteria. The European Commission has presented several suggestions of specific indicators for innovation and sustainability as well as training supporting materials (Directorate-General for Environment, 2016, 2019; Dodd et al., 2016). For the studied type of contract, the following suggestions can be given: 1. Innovation: better levels of usability, portability and interoperability with other health or communication instruments. 2. Sustainability: higher level of incorporation of recycled materials, lower energy consumption, lower pollution emission and higher percentage of obsolete equipment sent to recycling.

The Proposed Tree-Structure Model
The evaluation model can be summarized by the following evaluation presented in Figure 2 using the example of benefits mentioned for health services. The proposed descriptors for each criteria and sub-criteria are presented in Appendix A and they are assessed through a scoring scale representing the growing preference from 0 to 10.

The Quality of the Procurement Procedure
The Formation of the Contract with e the weighting coefficients (0 ≤ e ≤ e ∑ e ) Therefore, these two dimensions can be combined into a single indicator on the quality of the procurement procedure: with the weighting coefficients (0 ≤ ≤ e ∑ )

The Quality of the Service
with g the weighting coefficients (0 ≤ g ≤ ∑ g )

The Contract Cost
The indicator CB was already defined in 4.4

The Contributions to Innovation and Sustainability
The indicator IN descriptor is defined in Appendix A.
According to the MAUT theory (Dyer, 2016;Fishburn, 1970) an aggregated indicator of procedural and service quality can be now defined: being IS the global quality and h 1 h e h 3 the weighting coefficients (h 1 h h 3 ≥ 0 ith h 1 h h 3 ).
Consequently, the multi-criteria evaluation can be represented in a two-dimensional space relating the achieved quality expressed by IS with the cost performance measured by CB. Finally, a global value indicator, V (scale 0 to 10) will be defined by: being λ the weighting coefficients with λ 1 λ ≥ 0 e λ 1 λ .

The Estimation of Weighting Coefficients
The elaboration of the multi-criteria evaluation model requires setting up the weighting coefficients at three levels, for the four main dimensions (quality of the procurement procedure, quality of the service, cost performance and the contribution to sustainability and innovation), for the remaining criteria and sub-criteria. This topic has been extensively studied in Decision Theory (Tavares et al., 1997) and discussed by several authors suggesting alternatives approaches (Mousseau, 1992) and discussing the issues of applicability, vulnerability (Belton & Stewart, 2002), interpretation (Borcherding et al., 1991) and friendliness (Riabacke et al., 2009). Several approaches can be used: 1. Setting up an arbitrary initial set of values that can be improved iteratively through surveys, focus groups or the Delphi method; 2. More elaborated learning processes according to new methodologies, such as the "OPTIONCARDS" method (Maia & Tavares, 2013;Tavares & Arruda,2022).

On the Importance of Home Health Care Services
Health services are a growing sector in European Union (over 10% of the GDP) and more than 17,7% of the GDP in USA (OECD, 2019) and therefore new approaches to their management focusing on the quality of the provided service and increasing efficiency are a priority to cope with the increasing demand and the insufficient public budgets.
A major approach to cope with these new challenges corresponds to transfer health services from hospital centers to the patients home avoiding the high levels of congestion and costs which means that an hospitalization centered policy should be replaced by an home health care centered policy disseminating services to patients homes. There are many evidences (Landers et al., 2016;Liu et al., 2004;Vining & Globerman, 1999) showing that home services tend to be delivered more efficiently by private contractors and therefore a critical condition to implement this new policy is the ability of contracting such services with low transaction costs but, unfortunately, this objective has not been achieved quite often (Barthé lemy & Qué lin, 2006;Le Grand & Bartlett, 1993;Williamson & Chen, 2017).
Home health services have the nature of "incomplete contracts" already extensively studied in Economics (Hart, 2017;Hart & Moore, 1998;Martimort et al., 2005;Wang, 2016) because changes due to the progress of digital and health technologies and frequent changes of demand make the full specification of the contract object an impossible task. The impact of this incertitude is higher because the contracts duration should be multi annual so that the contractors will have the possibility of investing in human and technological resources. Thus, transaction costs tend to be high in incomplete contracts because the contractor gains the so called "residual rights" setting up very unbalanced conditions for negotiation between the contracting authority and the contractor whenever contract changes have to be considered. This explains the well-known problems of time and cost contract overruns tend to have quite a significant dimension.

The Case of Home Respiratory Services in Portugal
In Portugal, several shortcomings have persisted for many types of Home Health Care Services (HHC) awarded by hospitals and regional administrations and so the Government set up a Central Purchasing Body to avoid the lack of coordination or regional equity and to innovate the adopted procedures coping with the problems already presented.
An important class of HHC concerns the provision of Home Respiratory Services (HRS) which includes several types of services, namely ventilo-therapy, oxygen-therapy and aerosol-therapy, presenting a growing trend but also significant changes about the relative demand for specific services (Portugal Ministry of Health, 2019). The COVID19 pandemic created a new and disruptive increase in the demand for Home Respiratory Services, as in some of these services, namely, ventilo-therapy and oxygen therapy, the percentage of patients being treated at home increased, avoiding hospitalization and reducing the congestion of hospital units dedicated to the treatment of COVID19.
A centralized contracting procedure was initiated in 2008 using the traditional competitive procedure but extensive court litigation was generated and no award was achieved. In 2013, a joint research project lead by Observatory of Technology Foresight (OPET), the association of firms providing Home Respiratory Services, APCSD (Portuguese Association of Providers of HRS) with the support of the law bureau BAS developed an innovative procedure based on a framework agreement including all competitors accepting the contracting requirements and the minimal negotiated price for each type of service. Thus, a catalogue of all services and contracted providers was produced rather than awarding a contract to a single competitor and no litigation hampered the award decision and the act of contracting.
Therefore, every time a clinician prescribes a specific service to a patient he will have access to such catalogue and will select the provider to execute the service because there is a pre-selected set of providers respecting the quality requirements and accepting the same negotiated prices. This means that such providers are not competing in terms of the price but rather in terms of the quality to attract the preference of the final beneficiary who is the patient. Furthermore, the adaptation of the terms of the contract to new technological innovation or new patterns of demand is automatically imbedded by the providers into the execution of the contract because, otherwise, they will lose market share as the patients will prefer other providers and therefore the impacts of any residual rights are minimized.
The Portuguese Central Purchasing Body for Health (SPMS) has awarded the framework agreement in 2014 without any litigation or delays and this contract has been executed since then until 2018. Fortunately, this framework agreement was extended until 2019 and a new similar framework agreement was awarded in 2019 which is now being used helping to provide not just the usual services but also the new demand due to the fight against COVID19. In 2019, the presented model was developed to evaluate the contract execution involving SPMS, OPET, representatives of APCSD (namely, Maria Joã o Vitorino, Jorge Correia, Cá tia Caneiras, Paulo Caseiro and Jorge Huertas) and a panel of international experts (IPE). All these stakeholders and consultants gave important contributions to the structure of the model, to the estimation of the weighting coefficients and to the discussion of the obtained results.
The obtained results and their discussion are presented in the next section.

The Application of the Multi-Criteria Model to the Evaluation of an Home Respiratory Services Contract
The application of the proposed model implied the estimation of the weighting coefficients as well as all the presented indicators using the descriptions and their scoring systems.

The Estimation of the Weighting Coefficients
The adopted procedure included an extensive discussion held at a focus group including IPE as well as the members of APCSD and the authors of this paper in a total of 12 members. A consensual outcome of such discussion was obtained including the following weights: Given the similarity of all its dimensions, the hypothesis of neutrality was recommended: and considering the importance of the "patient-centered-care" (Porter & Teisberg, 2006)  The discussions carried out with SPMS and with APCSD have confirmed that this MCDA is able to embrace the multiple dimensions required to evaluate the contract performance of the Home Respiratory Services contract because it covers the formation of the contract, its cost, its contribution to sustainability and innovation as well as the quality delivered to final beneficiaries as perceived by patients and clinicians.
The obtained results from the application of the proposed MCDA are quite useful to support the classification and sorting decisions already explained: (a) The overall merit is about 8/10 which means that it belongs to the fourth quartile of this merit scale (b) The perspectives sorted for improvement are:  IC (6): duration of the procedure;  IC (7): clarity of the procedure documents;  IE (1): coherence of the manual of the execution guidelines;  IE (2): digitalization;  IN: contribution to sustainability and innovation.

Conclusions
The concept of contracting has not been subject to much attention by the authors of MCDA and DSS as the published results on contracting are just focusing on the pre-contracting stage namely, to improve the decisions of candidate's selection and bid evaluation. However, no applications were found about the complex issue of evaluating the overall performance achieved by the contract execution including the formation of the contract, its award and implementation as well as the produced range of benefits, costs and impacts.
This absence of such results is limiting the possibility of benchmarking the performance of contract execution across contracting authorities, countries or sectors and so the published results have a more descriptive and monographic nature missing the major added value of MCDA and DSS approaches which is the provision of a structured, formal and general mathematical tool allowing stable and general comparative results.
Therefore, the major contribution of this paper is proposing a MCDA model well adapted to the evaluation of the execution of public services contracts which produces results supporting two critical decisions: how much is the merit of the contract execution? and which aspects of the contract execution should be changed so that the contract performance will be improved?
A successful application of this MCDA model to the execution of a public services contract concerning Home Respiratory Services is also presented showing that its merit is in the fourth quartile of the overall evaluation scale and that the major areas for improvement are: (a) The too long duration of the procedure to form the contract; (b) The insufficient level of clarity of the procedure documents; (c) The insufficient of coherence of the manual of the execution guidelines; (d) The partial scope of digitalization; (e) The limited contribution to sustainability and innovation The proposed model can be now applied to evaluate other contracts concerning other contracting authorities and sectors allowing very useful benchmarking results to identify best practices, low performance cases and major areas for improvement.
Summing up, the presented model extends the potential of MCDA approach to the growing field of contracting allowing better, more formal, structured and stable instruments to evaluate the execution of each public services contract and their benchmarking across contracting authorities, sectors and countries.

A.1.1. The Formation of the Contract
The proposed descriptors for the formation of the contract dimension are represented in Table A1. It should be noted that these descriptors should concern the conditions set up by the contracting authority in relation to the market for contract award and so: (a) The accessibility should be described by how open the adopted procedure is to the participation of economic operators. Thus, for instance, if there is an open competition and just one bidder participates, this procedure is more open than if the contracting authority invites three bidders because in this case the acceptance of other bidders is rejected.
The descriptor transparency is focusing on how evident the adopted rational analysis is, leading to the award decision.
(c) The descriptor of negotiation should account for the existence or not of such type of interaction which can support better matching between buyer and supplier.
(d) The descriptor for litigation is measured by the delaying time because the real impact of litigation is the delay of the contract signature. For instance, in the case of having several litigants but being such disagreements easily settled down, then this situation is much better than having just one litigation taking too long to be settled down.
(e) The descriptor of stability should be measured by the time span for the service being contracted as service fragmentation into small service periods can be quite prejudicial (f) The descriptor of the duration of the process of contract formation is measured by the length time from start of the process up to the contract award.
(g) The descriptor of clarity should be based on the number of amendments introduced or requests for clarifications being received about the documents specifying how the contract should be executed after being awarded. The proposed descriptors for the contract execution dimension focus on the quality of the information specifying the terms of compliance for the contract execution (IE(1)-Coherence and Clarity ) as well as the degree of digitalization of the whole process for contract execution because the adoption of non-digital paperwork is a source for lack of quality and of time and resources waste. The degree of digitalization is expressed by the number of tasks (ordering, provision of the service, monitoring, reporting, billing, control, payment) executed through digital instruments avoiding paperwork. These descriptors are presented in Table A2. There is no manual presenting the execution guidelines.
No digitalization of operations of tracking, control, billing and payments. 2 The manual presenting the execution guidelines was published after the initial date of contract execution but has required clarification requests Partial digitalization of operations of tracking, control, billing and payments (1-20%); 4 The manual presenting the execution guidelines was published after the initial date of contract execution but has not required clarification requests.
Partial digitalization of operations of tracking, control, billing and payments (21-40%); 6 The manual presenting the execution guidelines was published before the initial date of contract execution but has required clarification requests which were answered after the initial date.
Partial digitalization of operations of tracking, control, billing and payments (41-60%); 8 The manual presenting the execution guidelines was published before the initial date of contract execution but has required clarification requests which were answered before the initial date.
Partial digitalization of operations of tracking, control, billing and payments (61-80%); 10 The manual presenting the execution guidelines was published before the initial date of contract execution and has not required clarification requests.
Partial digitalization of operations of tracking, control, billing and payments (over 81%); The degree of digitalization is expressed by the number of tasks (ordering, provision of the service, monitoring, reporting, billing, control, payment) executed through digital instruments avoiding paperwork.

A.2. The Quality of the Service
This indicator, IQ, should be based on the perception of beneficiaries and stakeholders. In the case of health services, the perception of quality of services should be estimated for the patients receiving such services (IQP -Table 3) and for the clinicians who prescribe and monitor the patients (IQC -Table A3). Describing the perceived quality being 0 extremely bad and 10 extremely good.

A.3. The Contributions to Innovation and Sustainability
The proposed descriptor is based on the recommendations of the European Commission (2019) for the innovation and sustainability dimension and it is presented in Table A4. The documents of the procedure do not require the specification of contributions to innovation and do not set up targets or requirements for sustainability 6 The documents of the procedure require the specification of contributions to innovation, but they do not set up targets/requirements for sustainability or the documents of the procedure do not require the specification of contributions to innovation, but they set up targets or requirements for sustainability 10 The documents of the procedure require the specification of contributions to innovation and set up targets or requirements for sustainability.

A.4. The Contributions Cost
As it was explained before, the cost performance was based on the savings compared to a cost reference level because the judgement about the merit of obtaining a specific cost depends always on its comparison with an alternative reference. Then, a transformation of such savings into the same scoring scale adopted to evaluate the other criteria should be adopted so that this performance will be expressed in the same scale. Ignoring such requirement has been classified by (Keeney, 1992) as the most common mistake of decision sciences. An extensive debate was held about such transformation to understand how "good" is obtaining a specific level of savings in http://ibr.ccsenet.org International Vol. 15, No. 3;2022 100 public contracting. This discussion was supported by the assumption that the savings unit interval can be adopted as 5% and by the classification of changes between 0 and 10% as minor changes (Tavares, 1999) (European Parliament and the Council of the European Union, 2014) implying that the higher levels (6 to 10) were reserved for savings higher than 10%. The proposed descriptors are presented in Table A5.

Copyrights
Copyright for this article is retained by the author(s), with first publication rights granted to the journal.
This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).