The Operation Effectiveness Evaluation Based on Statistic Analysis

Because there are many uncertain factors in the process of combat, the operation effectiveness data shows obvious randomness. In order to explore the guidelines of combat behind effects data, the article evaluates the effectiveness of attack which is a typical operation based on statistical analysis, and analyzes the basic concepts of operation and its effectiveness. For the simulation data produced by EINSTein system, the authors adopt the methods of once, single and multiple experimental analysis to study statistic characteristics of the effects data of attacks and find that. For the side who has advantages over the combat, improving firepower radius will have better effects than increasing the people who take part in the operation. On this basis, an evaluation method of operation effectiveness is put forward and verified combined with simulation data, and thus provides feasible solution to effectiveness evaluation based on practical training results.


Introduction
General carl von Clausewitz has said that war is an uncertain kingdom, and three quarters of factors that war is based more or less is surrounded by the fog of uncertain factors.(Hu, Xiaofeng, Si, Guangya, Wu, Lin, et al., 2004).In recent years, with the deep development of information war, continuous appearance of various new weapon and new technology has greatly increased uncertain factors of modern battlefields.In order to push aside war's fog of uncertain factors, military experts have attempted to explore the guidelines existing in war through operation effectiveness evaluation, have obtained many valuable research achievements (McCormick, Shon, 2010) and powerfully prompted the research and development of operation theory.
The problem of effectiveness evaluation has been always the hot topic in the field of military operations (Jiang, Jingzhuo, 2009).Operation is the core element in the process of battle; therefore, it is necessary to evaluate the effectiveness of operation firstly.In order to reduce the randomness of evaluation, (Peng, Zhengming, Li, Yunzhi & Luo, Xiaoming, 2005) used their own uncertain information to measure the degree that they finished the task and the difficulty was the confirmation of membership function; (Deng, Penghua, Bi, Yiming & Liu, Jifang, 2009) introduced ideal observer analysis method to solve the problem of uncertain decision-making effectiveness evaluation, but the method was based on static targets and lacked the discussion on dynamic targets in real war; (Xu, Shimeng, Xie, Gujin & Pan, Gaotian, 2000) carried out statistical analysis on Lanchester equation based on the probability; (Di, Changchun, Du, Zhonghua & Wu, Dalin, 2009) realized statistical evaluation on running smoothness of self-propelled gun through the method of statistical test.From this it can be seen that the method of statistical analysis have many advantages over dealing with random problems (Deng, Guilong, Liu, Zhihui & Jia, Zhidong, 2008) (Li, Ming, 2008), but currently the application in operation effectiveness evaluation was little.But the amount of operation data is often large, a feasible method needs to analyze the data and thus find out statistical guidelines existing in the process of operation.
Because there are too many factors that influence troop operation, and the effect data produced by operation in different conditions has random characteristics, the article adopts the method of statistical analysis to deal with operation effectiveness data to study their integral structure characteristics and their quantification relationship among data and then deduces general operation guidelines according to statistical characteristics and finally reach the goal of evaluating operation effectiveness.

Basic concepts
Operation is continuous activities which are taken for execute the task of combat.Currently, Chinese army defines it as basic fighting act which cannot be divided or is unnecessary to be divided in certain environment of battlefield.Accordingly, American Army gives the smallest atomic action in the process of battle abstractly-Action through the method of EATI template in Conceptual Model of Mission Space (CMMS) (Wang, Jian & Wang, Hongwei, 2009).
Although there are differences in the expression of two definitions, they are consistent in essence and highlight the characteristics of atomicity that cannot be divided any more.In order to discuss conveniently, the article takes the basic deeds as operation uniformly.In practical military problems, the characteristic that the operation cannot be divided any more is usually relative and it depends on the level of the problem that is studied.
Operation effectiveness is used to measure the contact ratio of operation effects and task demand and it is quantized by certain formula or algorithm, which means that effectiveness value is the function of effects and demand (Hu, Jianwen, Zhang, Weiming, Hu, Xiaofeng, et al., 2005).It can be seen that finally operation effectiveness depends on observable quantities of operation effects, the parameters that express demand and their method of comparison, and operation effects data are such observable quantities as maneuverable speed, enemies' injuries and death rate and so on.

Operation effect data statistical analysis
EINSTein system (Ilachinski A, 2011) is a combat simulation platform based on the combat simulation system ISAAC studied by Marine Corps Combat Development Command (MCCDC).The system is constructed based on Agent technology and provides new tools for studying war's complex problems such as uncertainty, nonlinearity and so on; therefore, based on EINSTein simulation data, from three angles including once simulation experimental analysis, single simulation experimental analysis, multiple simulation experimental analysis, the article discusses the simulation results in the scheme of multiple forces and weapons ratio in order to find the statistical guidelines existing in the process of combat in different conditions and lay good foundations for operational effectiveness evaluation.

Once simulation experimental analysis
Once simulation experimental analysis means to make an analysis on the data collected after once simulation running mainly, which mainly includes data presentation of simulation process and data statistics of simulation results.After obtaining simulation sample every time, both sides of operational results are statistically analyzed, in order to support next needs of evaluation.
Because there are many types of operations, for convenient analysis of the problem, the article mainly discusses attacks in unit level operation which is typical operation.Suppose that: the red side attacks the blue side, and occupies the position of blue side and wipe them out; the blue side has positional defense and repels the attack of red side.Initial setup of combat model: the map of battlefield is about 100*100; simulation time is takes 200 time units; the red side has 50 Agents; each Agent represents a individual soldier; according to the input conditions of EINSTein system, both sides' range of properties are confirmed as shown in table 1.
Figure 1 shows the situations of battlefield when the simulation time is 0, 100, 150, 200.
From the simulation process we can see that, the red side has overpowering advantages over wiping out all the blue side staff and occupies their position.Simulation results are both sides' damage situations: specifically, the red side remains 72 healthy people and it has 72 hurt people; the blue side has been killed in the moment of simulation 166.

Single simulation experimental analysis
Single simulation experimental analysis means to analyze the data of multiple simulation result in the condition that the simulation input parameter is fixed.When operation is combined by multiple random models, the experimental results produced by it will show a series of random data.Statistical analysis on the random data mainly includes following three aspects: sample numerical characters, interval estimation and hypothesis testing.
The setup of model parameters is the same as table 1.There are 100 times of simulation of attacks for the red side.Basic statistical analysis of simulation results is shown as table 2.
From table 2 it can be seen that, the sample variance of healthy people and hurt people of red side is more than the corresponding value of blue side, showing that in current model input parameter conditions, the fluctuation of the damage situations of red side is very large and its impact on final operation effectiveness is more obvious.The reason is that the effects of attack are mainly embodied in both sides' damage situations.The reconnaissance distance, firepower radius and desire to fight of red side in table 1 have obvious advantages, so the loss of blue side must be severe.This is shown by smaller sample variance for the blue side but larger variance for the red side.This shows the attack effects is mainly dependent on the damage of red side.From this one basic principle for operation is verified from the perspective of statistical analysis: for the side that has advantage over attacking, they should improve their survival ability in the operation so as to obtain better operation effects.
Suppose that each statistical result is normally distributed.In the condition that the confidence level is 95%, the normally distributed model parameters average  and variance  are given interval estimation.Although the assumption of normality is ideal from confidence interval, it is still necessary to carry out hypothesis testing further.
Q-Q figure is a directive method of evaluating normally distributed hypothesis, which shows the relation between samples fractile and observed value.When each point is very close to a straight line, the assumption of normality remains, otherwise, it is doubtable.The Q-Q testing for red side of statistical results is shown as figure 2.
From the distribution situations of data in figure 2, we can judge that the number of red side of healthy and hurt people is normally distributed whole, and similarly available, the corresponding damage data of blue side is also normally distributed.It can be seen that the hypothesis condition and testing result is consistent.Because each operation is independent mutually, and the initial parameters of the model is fixed, we can think that each simulation results are independent random variables with the same distribution and mutual deviation is small; therefore, it is reasonable for the hypothesis of simulation results to be normally distributed.

Multiple simulation experimental analysis
Multiple simulation experimental analysis means to analyze the data of multiple simulation result in the condition that the simulation input parameter is changeable.Through changing the initial conditions of operation, it studies the impact of different operational conditions on operation effects and thus supports the prediction of experimental results and evaluation of operation effectiveness.
Based on the setup of the parameters in table 1, the input parameters of blue side remain unchanged, and change the firepower radius and the number of total people of red side respectively; on the same way 50 times of simulation are carried out.The statistical results are shown in figure 3 & 4.
According to the output results of simulation, suppose that the number of healthy people of red side is 1 x , the number of hurt people of red side is 2 x , the number of healthy people of blue side is 3 x , the number of hurt people of blue side is 4 x , the total number of the red side is 1 N , the total number of the blue side is N 2 , then the ratio of the number that remains on red side is r  , calculate it through the formula (1): The ratio of the number that remains on blue side is b  , calculate it through the formula (2) It is formulated that the side will be defeated when the lost people reach 80%.The figure 3 shows the relationship of r  (corresponding to the left Y axis) and b  (corresponding to the right Y axis) of red side in condition of different firepower radius.For red side, their target is to try their best to attack their opponents when ensuring that their own damage is as little as possible, which reflects in the figure that the larger r  and b  is, the better.Through the regression analysis of the data of the results, it can be seen that with the increase of red side of firepower radius, r  increases slowly but b  increases rapidly, which shows that the impact of firepower radius on operation effectiveness is obvious and is mainly embodied in the attacking effects to blue side.Further analysis may find that when red side of firepower radius reaches 15, the loss of blue side has reached 80%, which means that they have lost ability to fight.With the increase of firepower radius, the loss of blue side continues to increases, but survival rate of red side doesn't increase, and higher injuries and death rate of blue side has little impact on operational result.In practical combat, human and material resources are often limited, so it is necessary to allocate various weaponry reasonably, which not only confirms the effects of attacking enemies, but also cannot blindly use high-level of weapons-of-mass, leading to the waste of resources; therefore, in above simulation conditions, it is reasonable for the range of firepower radius to be between 15 and 18.
Similarly, figure 4 shows the relationship of r  and b  of red side in the condition of different numbers of people in the combat.From the figure it can be seen that r  and b  are overlapped basically.In the same level, the survival rate of red side and the injuries and death rate of blue side do not increase obviously.Further analysis finds that with the increase of the number of people of red side, the survival rate of red side has no obvious increase, but because of the increase of the cardinal, the absolute quantity of injuries and death on red side increases in essence.This result is obviously not corresponding to the original intention that improves attack effects through increasing total number of people in the combat.The reason is that the initial conditions of simulation.Red side has obvious advantages over firepower radius and desire to fight than blue side.Under the premise, the number of people in the combat has little impact on operation effectiveness.It can be seen that no matter how many people red side has, the injuries and death rate of blue side is over 80% basically, which means that red side has reached their target to fight; therefore, for the side who has advantages over weaponry it should not be taken for granted that increasing the people in the combat will obtain good operational effectiveness definitely.Allocating staff appropriately and reducing our own loss in the condition of finishing the task of attacking enemies should be paid more attentions, in order to obtain ideal operational effects.
In the condition that red side has more advantages than blue side on the whole; the article above discusses the degree of impact of firepower radius and troop scale on operational effectiveness.According to the thinking method of research, in the condition that red side and blue side are very close or blue side has more advantages than red side, we can discuss the impact of various factors on operational effectiveness to find out some valuable operational guidelines, and thus provide powerful decision-making evidence for the department of command.

Operational effectiveness evaluation
The aim of the data statistics of operational effects is evaluating its effectiveness.In consideration of the characteristics that operation cannot be divided and the effects data is random, the evaluation process is shown in figure 5.
Operation happens in certain battlefield environment and is restricted by subjective and objective factors such as topography, armed forces, weaponry and so on, and operational effects data are the specific presentations under their influences.Operational effects data usually is a multidimensional vector shown by . Each dimension of the vector reflects the execution effects from one side.Considering of the characteristic of operational atomicity, it is necessary to map the multidimensional vector to the effectiveness index I which can comprehensively reflect operational effects: ( ) I f X  . Based on statistical analysis on effects data, the statistical characteristics of effectiveness index are obtained.Combined with the target of operational task, the demand effectiveness index req I is defined.By comparison of the coincidence degree between practical effectiveness index and demand effectiveness index, operational effectiveness value E is calculated out: ( , ) pra req E F I I  .Taking that red side attacks blue side in the article as the example, the basic process of operational effectiveness evaluation is presented.The impact factors are four input parameters of simulation: reconnaissance distance, firepower radius, desire to fight and maneuverability speed; effects data is the output of simulation: the number of the healthy people of red side is 1 x , the number of hurt people of red side is 2 x , the number of healthy people of blue side is 3 x , the number of hurt people of blue side is 4 x , then effectiveness data vector 4

X R 
. X reflects the operational effects of attacks from the aspects of red and blue side; accordingly, the effectiveness index I is defined as the ratio of damages between two sides, and is specifically shown by formula (2): 100, 50 represents the total number of red and blue side respectively.100 times of data of simulation results in section 2.2 is adopted to make statistical analysis to confirm effectiveness index I satisfies the normal distribution with the average 0.2311 and variance 0.0655, whose histogram is shown in figure 6.
According to the distribution of effectiveness I and operational demand of attacks, 0.3 req I  is confirmed and thus operational effectiveness value is defined as: After confirmation of the formula of operational effectiveness value, the effectiveness evaluation of attacks can be carried out.Supposed red side attacks once, according to the formula (3), the ratio 0.253 of damages between two sides is obtained from effects data.Then according to the formula (4), the operational effectiveness value 0.843 is calculated out, and we can think that the probability that red side finishes this attack successfully is 0.843.

Conclusions
The simulation data based on EINSTein system adopts the method of statistical analysis to do research in attacks which is typical operation in depth.The statistical guidelines existing in the process of combat is found out from repeated simulation experimental data.Based on the analysis of effectiveness index probability distribution, operational effectiveness evaluation method is put forward.Although simulation data is adopted, the method of data statics analysis and the thought of operational effectiveness evaluation have high reference value for the research of practical operational effects data.
Accurate evaluation cannot do without reliable data; therefore, the collection of practical operational data should be paid more attention to do research on the problem of operational effectiveness evaluation is studied combined with practical data of troop maneuver on the basis of simulation data in the article.In addition, in multiple simulation experimental analysis, the operational guidelines influenced by single factor is discussed and the impact of double factors and multiple facts are not analyzed; the operational effectiveness is evaluated under the same impact factors but is not discussed under different impact factors, which are the emphasis of further research work.Simulation and Training. Lecture Notes in Computer Science, 2009, 5551(1), 946-955. http://dx.doi.org/10.1007/978-3-642-01507-6_107 Xu, Shimeng, Xie, Gujin & Pan, Gaotian. (2000).The Basic Discussion about the Numerical Simulation of a Campaign Model.SYSTEMS ENGINEERING-THEORY & PRACTICE, 2000, (7), 106-111.[80.0497,81.9703] [16.6770,18.5030] [2.8705,3.7095] [3.9799,4.9801] [4.2494,5.6223] [4.0401,5.3454] [1.8564,2.4562] [2.2131,2.9281]

Figure 1 .
Figure 1.Schematic diagram of the simulation process of attacking

Table 1 .
Scope property