Consensual Experts ’ Opinion in Forecasting

In forecasting rental levels, different experts may have different opinions. Subjective expert forecasts would be useful but the platform for discussions may not be readily available let alone mechanism for reconciling the differences of opinions. The viability of aggregating and pooling anonymous opinions of experts in making forecasts of rental levels of purpose-built office space in Kuala Lumpur was investigated by empirical work. The Delphi technique was utilized to study the extent of movement of opinions towards consensus, and validation of answers by experts. The experts included in the Delphi process were private valuers/consultants, Government valuers and economic planning unit officers. The actual rental level was compared with experts’ pooled opinion through the technique. The technique adopted in this research had provided an instrument for experts to arrive at a consensual opinion on future rental level of office space in Kuala Lumpur.


Introduction
The aim of the study presented in this paper was to analyse how expert valuation opinions in making rental forecast could be aggregated.The well-known Delphi technique (Dalkey, 1969(Dalkey, , 1975) ) was utilized.The main substance of this technique is to obtain a consensual average opinion of individuals through a series of intense questionnaires sent separately to the individuals and the interaction is facilitated through controlled feedback (Dalkey, 1969, p. 408).
The technique which is claimed to avoid "biasing effects of dominant individuals" (Dalkey, 1975, p. 408) has been used with success in many areas involving judgmental forecast.

Social-psychological dynamic of behaviour
In order to meet the requirement of handling pressure for conformity, dominant personalities and low status member deferring his/her opinion, the Delphi technique was applied specifically for aggregating opinions of experts from a wide background.
The anonymous opinions aggregated and pooled using the Delphi technique, were related to market rental forecasts of purpose-built office buildings in Kuala Lumpur in the first quarter of 2008.
The experts included in the Delphi process were private valuers/consultants, Federal Government valuers from the Department of Valuation and Property Management, and experts related to forecasting i.e. economic planning unit officers.They were chosen based on their involvement with the property market related to rental movements.The majority of the experts however, were the private property valuers/consultants as they formed the group with the most direct contact with market transactions.
In general, the anonymity of opinions was essential here as the individual experts in the market forecasting of rental would be expected to be "sensitive" to his/her opinion as the forecast would either be proven right or wrong eventually.Disclosing one's opinion to other experts was expected to jeopardize one's professional integrity if the opinion is proven "inaccurate".

The Delphi technique
The main objective of the technique is to achieve anonymity among a group of experts.Linstone and Turoff (1975, p. 4) suggest that a structured communication procedure provided by Delphi is suitable where "Disagreements among individuals are so severe or politically unpalatable that the communication process must be refereed and/or anonymity assured".
The Delphi technique which was developed by Norman Dalkey (1969) and his associates at the Rand Corporation essentially involves pooling of experts' judgements (normally five or more).In the business world where it was originally developed and effectively implemented, it aims to forecast increases or decreases of business sales.
Basically, the process starts with each individual expert separately (using a questionnaire) being requested to give his subjective opinion on the probability (numerical) of occurrence of an aspect under investigation e.g.future sales of a given product of a company.The opinions are then collected and analysed in the form of a summary of basic statistical description (i.e.mean, median, standard deviation and range of the opinion of the world group) and passed back to the personnel for reconfirmation or a review of his/her opinion based on the summary of group opinion.This process may continue for several rounds until a consensus opinion is obtained.This, however is not expected to last more than two rounds in most cases (Jolson and Rossow 1971).
The most crucial aspect of the technique is validating the results (Jolson and Rossow, 1971).The methodology of validation has been to relate the opinion of the personnel to the level of confidence in giving an opinion (Best, 1974) or providing extra questions which are outside the specialization (or nature of the profession) of the personnel testing them against additional samples most likely to be able to provide opinions on the additional questions (Jolson and Rossow, 1971).
In this research, forty consultants from all the main property consultant companies, the rating valuers and economics planning unit officers from the City Hall of Kuala Lumpur with different levels of experience in the property market in Kuala Lumpur have been included in the sample.Each individual consultant was given a covering letter about the Delphi process and a questionnaire containing three basic questions.The cover letter and questionnaire were sent to them in September 2006.
The first question requested the consultants to provide their opinion on the probability that certain levels of rental of a certain class of purpose-built office complexes within a well-known defined location in Kuala Lumpur City will be reached in about one and a half year's time.Two other questions in the questionnaire asked for the opinions of the consultants on aspects related to shopping complexes and purpose-built office complexes (where the actual answer was known) respectively for validation purposes.For each question, the consultants were requested to provide their level of expertise in giving their opinion and the reason(s) for their opinions.The consultants were also requested to state the length of experience they have in the property market.

Analysis of opinions involving a large number of personnel (the Delphi technique)
During the course of the knowledge elicitation, it was found that there was a need for aggregating anonymous opinions of a large pool of experts.
In a most comprehensive book on the Delphi technique edited by Linstone and Turoff (1975), one of the reasons stated by the editors for the failure of a Delphi technique is that there are difficulties of analyzing the results i.e. analyzing the disagreements between the experts.Interpretation and summation of responses are never complete, because the panel of respondents does not send in their "heads" but only their responses.A complete record of the interactions is unwieldy and diffuse and difficult to use (Scheele, 1975, p. 70)

Requirements in a Consensual Technique
In the process of analyzing the pooled opinions using the Delphi technique, Murray and Turoff (1975, p. 88) state that the researcher should be clear on the following aspects: (a) Issue or issues that need consensus.
(b) Analysis of the opinions of the experts namely, i.The opinions and reasons given by the experts and the extent of their opinions moving towards a consensus.
ii.The comparison between the experts on a relative basis.Here, the following potential effects of personal variables on the participants' (experts') behaviour should be taken into account:-Background of experts depending on the purpose of the research.
Confidence/self-rating of experts (Linstone and Turoff, 1975, p.234) i.e. how confident or certain are the experts (Scheibe et al, 1975, p. 263) Effects of the confidence of the individual experts on the movement of their opinions towards consensus (Scheibe et al, 1975, p. 274) The form(s) of the feedback to the opinions (e.g.average/median/standard deviation etc.) How does the group view and re-evaluate the separate arguments used to defend various positions or opinions?This include the following questions: Does the sharing of the different reasons affect their re-evaluation of opinion?How stable are the opinions between rounds?(c) How to validate the opinions?

Technique adopted
The technique adopted fulfil the requirements above can be said to be related to the design of the Delphi questionnaire in the process of obtaining the subjective opinions of a pool of experts comprising the private valuers (majority) and other experts.
Due to the considerable list of requirements, for convenience, the requirements are listed first, followed by the techniques adopted to achieve them.

Clear on the issues that need consensus
In order to ensure clarity of the issues that need consensus, a cover letter and specific questions were included.

Cover letter
A cover letter was included with every questionnaire.In the round 1 questionnaire cover letter, the objective of the questionnaire was clearly stated to the experts.The objective that was stated was pooling the subjective opinions (i.e.making judgmental forecasting in matters pertaining to commercial property) using the Delphi technique.
The Delphi technique, involving what the experts really have to do as participants, was also outlined in the cover letter.
In round 2, the cover letter comprised recapitulation of the Delphi technique and the contents of round 2 questionnaire comprising summarized opinions of round 1 and request for experts either to defend their opinion in round 1 or review their opinion.

Specific questions
In round 1, the questions asked in the Delphi questionnaire were made specific following Scheibe et al. (1975, p. 263).At the outset, it was decided that the opinions that needed to be pooled should be related to the subjective market forecasting of rentals (related to market knowledge) of major commercial properties.
Prior to deciding the actual questions to be included in the questionnaire, discussions were held with the core valuers and two very experienced and currently practising private valuers in Kuala Lumpur City to decide suitable specific questions.
Based on the discussions, it was decided that three questions relating to opinions on aspects of office and shopping complexes were to be included in the questionnaires.
The questionnaires were sent to the experts at the beginning of September, 2006.

6.1.2(a) Question 1 (Core question)
The experts were requested to state their subjective estimate of the probability of various range average rental levels per square foot ( per month) of one class of purpose-built office space in the Golden Triangle Area of Kuala Lumpur in the first quarter of 2008.The various range average rental levels per square foot (per month) were decided based on the discussion with the core valuers and the two very experienced private valuers.
To guide the experts, examples of offices falling under the class were given.In addition, the average rental of the offices in the third quarter of 2006 was also provided.

6.1.2(b) Question 2 (Validation for Question 1)
The experts were requested to state their estimate on the total supply of purpose-built office floor space in Kuala Lumpur City as of December, 2005.This is a validation question for Question 1. since the approximate actual answer for Question 2 was known, the extent of divergence of the aggregate experts' answer for Question 2 will serve to check whether their answer for Question 1 was valid.In other words, Question 2 aimed at validating the experts' answer for Question 1.If the aggregate answer moves towards the actual known answer, the movement towards a consensual answer (for question 1) would be theoretically validated.

6.1.2(c) Question 3 (Another validation question for Question 1)
The experts were requested to state their estimate on the total supply of retail space in shopping centers in Kuala Lumpur City as of December, 2005.This is another validation question for Question 1.

Analysis of opinions of the experts
The analysis of opinions of the experts involved scrutiny of the followings:-

Opinions and reasons given by the experts and the extent of their opinions moving towards a consensus
Besides seeking the opinions, the experts were also requested to state the reasons for stating their subjective opinions.This was aimed to provide guidance to the analysis of the subjective opinions of each expert.In addition, the reasons can be fed back to the experts in a subsequent round of the exercise.
The extent of their opinions moving towards a consensus (Jolson and Rossow, 1971) was analysed based on the followings:-(a) Direction of answers; this involved studying whether the opinions were moving towards the same direction.
(b) Range of answers in each round; this involved studying whether the range of opinions get smaller as the rounds proceeded.(c) Analysis of variance between rounds; this involved analyzing whether the variance in round 2 gets smaller compared to round 1.

Comparison between experts on a relative basis
The opinions of each individual expert were examined at each round of the questionnaires.

Background of experts depending on the purpose of the research
The experts were asked to sate the number of years they have been involved in the property field.In addition, they were also asked to state the number of transactions in commercial properties expressed in terms of Ringgit Malaysia (RM) that they had undertaken.These aimed at analyzing the opinion of the experts in relation to their deduced level of experience in the commercial property field.

Confidence / self-rating of experts
In each round of the questionnaires, the experts were asked to sate their self-rating level of expertise when answering each of the questions in the questionnaires.This follows Dalkey (1975, p. 246) who states that a visible index of certainty is self-rating i.e. judgement by the individual of his or her competence or level of knowledge concerning the estimate.
Traditionally, this takes the form of requesting the experts to state their self-rated expertise in the form of a scale.Scheibe et al. (1975, p. 267) indicates that the scale chosen should be easily understood by the participants.In this research, based on the discussion with the core valuers, a scale of 0 (lowest self-rated expertise) to 10 (highest self-rated expertise) was adopted.

Confidence of the individual experts and adherence of their answers in the first round
The relationship between the experts' level of confidence and the extent of their second round answer to move towards consensus was considered.

Forms of feedback to the opinions
To provide each expert with an overall view of the opinions of other experts in round 1 of the questionnaire, the general summary of the opinions (which were passed to the experts in round 2 of the exercise in January, 2007) contained the following items.This includes the general and specific reasons given by the experts in round 1.For example, a general reason given by an expert for a positive forecast in Question 1 may be "good demand for class B offices" whilst the specific reason may be due to the effect of the Visit Malaysia Year in 2007 and the implementation of the projects related to the Ninth Malaysian Plan (providing good demand for business and consequently office space).

Distribution of the self-rated expertise of the experts
Distribution of the self-rated expertise of the experts in answering a particular question was summarized in the form of a histogram.This provided an overview of the experts' confidence when answering a particular question in round 1 of the questionnaire.It aimed to allow an individual expert to compare his/her own self-rated expertise with the overall self-rated expertise of other experts in the pooled opinion, thus, providing him/her information on his/her relative standing when defending or reviewing his/her opinions in round 1.Such an approach is supported by Scheibe et al. (1975., p.270).
Based on the summarized feedback, in the round 2 questionnaire, each individual expert was requested to review or defend his/her opinion in round 1.
At the outset, it should be noted that the probability medians for answers given to question 1 (i.e.Various Average Rentals of Class B Office Space in Golden Triangle Kuala Lumpur in the 1 st Quarter of 2008) were first normalized so that the sum of the probability medians totaled one.An example of the normalized probability medians for round 1 is shown below in Table 1.

How does the group view and re-evaluate the separate arguments used to defend various positions or opinions?
This question should be detailed out as follows.
(a) Does the sharing of the different affect their re-evaluation opinion?(b) How stable are the opinions between rounds?In this research, this involved analyzing the difference between each individual expert's opinion in round 1 and round 2. Issue b) however was not covered in this research.Only two rounds were undertaken preventing such analysis.This limited number of rounds was due to the difficulties of obtaining responses from the experts.This was due to the tight schedule of a number of experts when the second round of the exercise was undertaken making follow-up difficult.

How to validate the opinions?
Two forms of validation were undertaken as follows.
(a) Validating questions As stated under 1) above ("Clear on issues that need consensus"), two validating questions were included in the questionnaires.The results will be explained in section 8.2 below.
In addition, an objective testing was also undertaken.

(b) Objective testing
The actual rental level in the first quarter of 2008 was compared with experts' pooled opinion through the Delphi technique.The results will be explained in section 9 below.

Results of the Delphi technique
Only two rounds of the Delphi process were undertaken.Out of the 40 respondents initially identified when the process started, 36 were selected.The other 4 respondents were not selected due to technical errors in their answers to the questions in the first round.
For example, instead of giving answers to question 1 in terms of probabilities, they provided answers in terms of rate of increase in rental levels.

Analysis of the consensual opinions (a) Extent of movement towards consensus
The normalized probability medians for answers to question 1 (i.e.Various Average Rentals of Class B Office Space in Golden Triangle Kuala Lumpur in the 1 st Quarter of 2008) in round 1 and round 2 are shown below in Table 2.
(b) Direction of answers Looking at the median probability in both round 1 and round 2, the answers moved in the same direction.The rank of median probability for the various classes of the Average Rental prediction between round 1 and round 2 were the same.For example, the highest median probability for both round 1 and round 2 was the class of Average Rentals RM5.30 to RM 5.50 followed by classes RM5.55 to RM5.70, RM 4.80 to RM5.00, RM5.05 to RM5.25 and RM4.50 to RM4.75.It can also be observed that the first two highest median of probability in round 1 was "re-emphasized" in round 2.

Range of answers in each round
A movement towards consensus may be reflected by the range of the probability medians getting smaller as the rounds proceeded (Jolson and Rossow, 1971).The range of answers for round 1 and round 2 in this research is shown in Table 3.
As indicated in Table 3, there was no clear indication that the range of the probability medians getting smaller in round 2 compared to round 1 for the first four classes of Average Rentals of Class B office Space in golden Triangle Kuala Lumpur in 1 st quarter of 2008.
A closer study of the data however revealed that the absolute answers of "probability of 1.00" for the first three classes came from four very senior valuers in the sample with experience of between 15 and 31 years in the property field.
To a certain extent, it may be argued that there was a certain degree of 'inertia' or 'strong adherence" in their answers which may reflect their very strong confidence when giving their opinions.This very strong confidence of the valuers was supported by the very high self-rated expertise given by them i.e. 8 and 9 (out of maximum 10).
Quit apart from the absolute "probability of 1.00" answers on the first three classes of predictions above, there was to a certain extent, a tendency for the narrowing of range of answers for a class of prediction i.e.Average Rentals of RM 4.50 to RM 4.75.There was also no widening of range of answers for the class of prediction of Average Rentals of RM5.05 to RM5.25.These provided indications for a certain degree of consensus among the respondents regarding the rental forecast.

(c) Analysis of variance between rounds
In comparing the variance between rounds, the sample space may be considered to be the set of respondents' possible probability estimates.If the variance values of the two rounds are expressed by sd1^2, sd2^2, for there to be an indication for the movement towards consensus, the actual variance for round 2 should be getting smaller compared to round 1.
The findings from this research based on the analysis of variance are stated in Table 4.
As can be seen from Table 4, in general, the initial round showed widespread individual answers, but with feedback in round 2, the distribution of individual responses narrowed.By the same token, there was a tendency towards narrowing down of answers between the respondents in round 2.
However, the F-ratios between the two rounds accepted the null hypothesis i.e. the difference between the answers in round 1 and round 2 was not significant.Although the respondents tended to move towards consensual answers in round 2, in general, they had not in fact changed their answers in this round substantially as reflected by the F-values.25 out of the 36 respondents (69.44%) did not change their answers in round 2. This phenomenon could indicate the issue of number of rounds of estimation needed to reach consensus.
In this research, it could be argued that since the number of rounds was limited (only two), the effects of the feedback was not strong enough to change substantially the opinions of the respondents.However, although this could be a possibility, the fact that there was no significant difference in answers between round 1 and round 2 could also be an indication that the consensus had been reached as early as round 2. A substantial number of rounds of estimations is not essential in most cases, nor is it a prerequisite -studies have shown essentially no significant change after the second round of estimation (Best, 1974, Dalkey, 1969).A non-significant difference between rounds could indicate that the opinions were stable and hence consensus could have been achieved (Jolson and Rossow, 1971.)Moreover, it was found that the reasons given by the respondents when answering question 1 in round 2 were almost similar to the reasons that they had given in round 1.More interestingly, there was no real difference in the "spirit" nor "tone" of the answers between the respondents.Except for one certain detailed issues namely the possible impact of a specific event -"the Visit Malaysia Year" to be held in 2007 in Malaysia that will affect the demand for more office space, almost all respondents indicated rather similar factors that will affect the supply and demand for office space in Kuala Lumpur.This could be expected as the respondents were all actively involved in the property market in Kuala Lumpur.In addition, the opinions of the respondent were based on past history of the market as opposed to the pure Delphi scenarios where there is limited past history on the subject that is being forecast (Best, 1974).All this could point to the possibility of consensus being achieved (at round 2) without the need to pursue further rounds.However, it would be equally interesting for further research to confirm this by undertaking further rounds of estimation.

Validation of answers by respondents
For the purpose of the answer to question 1, the answers to the validation questions of 2 and 3 should move in the direction of the "correct" answers which were known in this research.This could be analyzed from the median and range of answers or estimates by the respondents in each round.A movement towards consensus would be indicated by a movement of the median towards the "correct" answer and decreasing size of range of estimates among the respondents Table 5 below shows the findings in this research in respect of question 2 and question 3.
Table 5 indicates that the median of the answers moved towards the "correct" answer in round 2 for the question on the estimation of the supply of office space.For the question on the estimation of supply of retail space, the median in round 2 remained the same with the median in round 1 reflecting the stability of the answers which in both cases were already quite close to the 'correct" answer.The ranges for both question 2 and question 3 were narrowed down in round 2. Based on the responses of the respondents to the validation questions 2 and 3, it could be positively deduced based on the Delphi technique that the respondents" answers to question 1 was converging towards a 'correct" answer.
In addition, the assumption that the "correct:" answer will continue to be an underlying force, even under the obvious attraction of the group median (Jolson and Rossow, 1971.) was supported.For example, more than half of the responses to the round 2 questionnaires for the validation study were closer to the "correct" answer than to the median returned from round 1.

Objective testing: Extent of accuracy of the consensual opinion
The average rental for Class B office complexes in Kuala Lumpur in the first quarter of 2008 was RM5.50.
The aggregate consensual opinion of the respondents thus proved to be a correct market rental forecast.

Analysis of the opinions between the experts (a) Experts' reasons for opinions
In general, it was found that not only were the answers of the respondents moving towards a consensus, there were also very strong similarities between their answers.The similarities of answers were observed to occur in both rounds 1 and 2. In addition, there was also a tendency for some valuers to appreciate the reasons provided by other valuers.In other words, there was an evidence of sharing of reasons among the valuers.

(b) Background of experts
The issue here is whether there were any relationships between years of experience and the effects of the feedback on their opinions in the subsequent round (round 2) Logically, it could be expected that experts with very long experience would stick to their original opinions and not easily be affected by the feedback from the process.However, based on the results, it was difficult to see whether there was any relationship between the length of experience the experts had and how much they were affected by the feedback given to them.It was observed that even respondents with very long experience changed their mind in round 2 and a number of respondents with short experience stuck to their original opinion.For example, two respondents with 17 and 26 years of experience respectively changed their opinions in round 2, whilst three respondents with only three years experience adhered to their original opinions.

Conclusion
Overall, the Delphi technique adopted in this research had provided an instrument for experts to arrive at an averaged consensus (Best, 1974) on future rental level of office space in Kuala Lumpur.
Using standard descriptive statistics in conjunction with thematic analysis and the number of comments generated, it was possible to demonstrate movement towards consensus and stability in this Delphi study.Following the original use of Delphi in social science, Delphi is suggested to be an effective way to gain and measure group consensus in rental forecasting of purpose-built office space.There is potential here to add clarification to the use of a very subjective methodology.It is suggested that a combination of the simple descriptive statistics as presented here be used to reduce subjectivity and ensure maximum validity of results in Delphi methodology for improved evidence of consensual decision-making.The trends observed in this exploratory study suggest that a larger study is warranted, following the same approach.
7.3.1 General summary of opinions of the experts in round 1 (a) Median of the opinions of the experts (b) Lowest opinion (c) Highest opinion (d) The actual opinion of the respective expert 7.3.2General logical explanation given by the experts in round 1

Table 1 .
Normalised probability medians for Question 1 in Delphi technique

Table 2 .
Medians for answers to Question 1 Delphi technique

Table 3 .
Range of answers for round 1 and 2 of Delphi Technique

Average Rentals of Class B Office Space in Golden Probability median Triangle Kuala Lumpur in 1 st Quarter of 2008 Round 1 Round 2
the largest and smallest numbers in the set of answers are given in parentheses beneath the range. *

Table 4 .
Analysis of variance between rounds of Delphi technique

Table 5 .
Median/range answers for validation of Delphi technique

on total supply of retail space in shopping centres in Kuala Lumpur as of December 2005*
*All answers are in millions square feet