Boltzmann Machine and Hyperbolic Activation Function in Higher Order Network

For higher-order programming, higher-order network architecture is necessary to provide faster convergence rate, greater storage capacity, stronger approximation property, and higher fault tolerance than lower-order neural networks. Thus, the higher-order clauses for logic programming in Hopfield Networks have been considered in this paper. The goal is to perform logic programming based on the energy minimization scheme is to achieve the best global minimum. However, there is no guarantee to find the best minimum in the network. However, by using Boltzmann Machines and hyperbolic tangent activation function this problem can be overcome.


Introduction
Neural Networks is a mathematical model or computational model that is inspired by the structure of biological neurons such as the brain process information.It can solve sophisticated recognition and analysis problems.It is because it composed of huge amount of interconnected neurons to solve specific problems.However in this paper, we are concentrated on Hopfield network.Hopfield network is a recurrent neural network (Hopfield, 1982) invented by John Hopfield, consists of a set of N interconnected neurons which all neurons are connected to all others in both directions.It has synaptic connection pattern which involving Lyapunov function E (energy function) for dynamic activities.It serves as content addressable memory systems with binary threshold units.
Logic is deals with true and false while in the logic programming, a set of Horn clauses that formed by atoms are represented to find the truth values of the atoms in the clauses.It is using neurons to store the truth value of atoms to write a cost function for minimization when all the clauses are satisfied.In addition, a bi-directional mapping between propositional logic formulas and energy functions of symmetric neural networks had defined by Gadi Pinkas (Pinkas, 1991(Pinkas, , 1992) ) and Wan Abdullah (Wan Abdullah, 1991, 1992).Further detail can refer to the references.The advantages by using Wan Abdullah's method are it can revolves around propositional Horn clauses and learning ability of the Hopfield network and hunts for the best solutions, given the clauses in the logic program, and the corresponding solutions may change as new clauses added.This paper is organized as follows.In section 2, an outline of Hopfield network is given and in section 3, method of doing logic programming in neural network is described.Meanwhile in section 4 contain discussions regarding the Boltzmann machine and Hyperbolic Tangent activation function.Finally, section 5 and 6 occupy the simulation results and concluding remarks regarding this work.

Higher Order Hopfield Networks
Discrete Hopfield network is shown in Figure 1, as an expanded form of a common representation of the Hopfield network.Hopfield had stated that this network is useful for solving combinatorial optimization problems as a content addressable memory or an analog computer.Combinatorial optimization includes looking for the combination of choices from a discrete set which produces an optimum value for some related cost function.In neural network, higher order logic programming is highly regarded as the essential method in Hopfield Networks.When the Hopfield neural network is used to solve NP-complete optimization problem (Brenton, 2002;Ding et al., 2010;Cheung & Lee, 1993) such as travelling salesman problem, positive solutions would be produced.From literature review, few papers had carried out that applying higher order Hopfield networks such as using HOHN (Ding et al., 2010) to solve N-queens problem and construction method of energy function and neural computing method also shown.Besides, comparison with the first order Hopfield network and the method how to speed the convergence and escape from the local minima also had discussed in those papers.While according to Cheung and Lee (1993), the convergence property had restudied before put in application in real life.Besides, Ising spin problem also had carried out in it.The most important paper that affects the main backbone of this paper is explained by Joya et al. (2002).It is a study of the different dynamics in HOHN and problem affecting practical application of these networks are brought to light such as incoherence between the network dynamics and the associated energy function, error due to discrete simulation on a digital computer, existence of local minima and convergence depends on coefficients weighting the cost function terms.However, in this paper only local minima and convergence are concentrated.Further explanation can refer to reference.From those stated above, Hopfield network has overcome the difficulty to find suitable parameters to guarantee convergence and explore a new path for artificial intelligence and intellectual computer.
Other than that, higher-order Hopfield network can solve non-linear and discontinuous data in larger field and connections.For example, it is well performed in nonlinear statistical modelling and it can provide a new alternative to logistic regression in bigger state and numbers.Furthermore, it is able to detect all possible interactions between predictor variables such as detect complex nonlinear relationships between dependent and independent variables.Lastly, it can be used as research tool like neurobiologists use it for interpretation of neurobiological phenomena.From here, researchers know that Hopfield network can use to minimize a configurationally energy function and thus can solve the combinatorial optimisation problem.It is a reason why there are good solutions can be found.
The higher-order Hopfield Networks with the order = n-1 is stated as below.The energy function is where is the state value of neuron , defines the connection weights of the nth order connection from neurons … to neuron I, is the input potential to neuron i and is the state of neuron i.In the high-order model each node is assigned a sigma-pi unit that updates its activation value by first computing the partial derivative of the energy function.The dynamic equation or the updating rule of the network is ), where sgn is signum function.The connection weight of higher-order Hopfield networks is symmetrical.This condition is analogical to the symmetric requirement of the Hopfield Netowrks connection weight matrix.
As the dynamic equation is derived by partial derivative, it only guarantees convergence towards local minimum.
It is affected by the value of the coefficients that weight the different terms of the cost function (Joya et al., 2010).Thus, the Boltzmann machine and Hyperbolic Tangent activation function will carry out in this paper.

Logic Programming
Logic programming is the use of mathematical logic for computer programming.Thus, higher order Hopfield network (HONN) had carried out in logic programming model.A HOHN is used to minimise logical inconsistency in interpretations of logic programs and clauses.To apply it, first of all need to understand what logic program play in the system.Following is the logic program that built by using Wan Abdullah's method in HN.
Following is the algorithms.
i) Given a logic program, translate all the clauses in the logic program into basic Boolean algebraic form.
ii) Identify a neuron to each ground neuron.
iii) Initialize all connections strengths to zero.It assumed the connection with A, B and C is zero value.
iv) Derive a cost function that is associated with the negation of all the clauses, such that 1 represents the logical value of a neuron A, where is the neuron corresponding to A. The value of is define in such a way that it carries the values of 1 if A is true and -1 if A is false.Negation (neuron A does not occur) is represented by, 1 ; 1 1 1 … … a conjunction logical connective 'and' is represented by multiplication whereas a disjunction connective 'or' is represented by addition.v) Obtain the values of connection strengths by comparing the cost function with the energy, H which in the section 2 that had recognized in Hopfield network.
vi) Let the neural networks evolve until minimum energy is reached.The neural states then provide a solution interpretation for the logic program, and the truth of ground atom may be checked then consider the solution obtained is a global solution or not.
A logic program contains of program clauses and it is activated by an initial goal.It is easy to understand, modify and verify.For example in a simple propositional case, logic clauses had formed as , , , … … , where the arrow can be read as 'if' while the comma can be read as 'and' for the purpose of interpretation the clauses by using truth value.Thus, a model or pattern can be found to the given logic program and it can be a way to solve the combinational optimization problem.Consequently, to carry out a logic program, we need to build up a simulator to run it.However, to solve the global minima problem, in next section, an introduction about Boltzmann machine and Hyperbolic Tangent activation function will carry out.

Boltzmann Machine
Hopfield networks have recognized that some relaxation schemes have a joined cost function and the states of the network converge to local minima of this function.It had performed optimization of a well-defined function.However, there is no guarantee to find the best minimum in the network.Thus, Boltzmann Machines had introduced to overcome this problem.A Boltzmann machine is a network of units which are fully interconnected by bidirectional connections with symmetric weights.There are no self-connections are allowed.These units have binary values 0 and 1 that referring to states ON and OFF for each unit.The major difference from Hopfield networks is the way of updating the states which determined by stochastic decisions.Boltzmann machines have a simple learning algorithm that allows them to discover interesting features that represent complex regularities in the training data.Furthermore, it extends the concept of Hopfield networks by a stochastic update method.
A Boltzmann machine, like a Hopfield network, is a network of units with an "energy" defined for the network.Unlike Hopfield networks, binary units of Boltzmann Machine are stochastic.By referring to the energy function, E (u) for Hopfield networks, due to the probabilistic update rule, a Boltzmann machine is able to transit the states on higher energy level in contrast to Hopfield network.This feature can avoid the network getting stuck in local minima of the energy function in minimization problems (Sathasivam & Wan Abdullah, 2010).The difference in the global energy that results from a single unit i being 0 versus 1, written  E i , is given by: www.ccsenThus,  E with u i = 1 given by: where the unit and se of the netw the probab from the n temperatur minimum.gradually the molecu initial posi whereas th   same ratio value when run for fifth order clause).To help to get through the problem of local minima, Boltzmann machine and hyperbolic tangent activation functions are implemented to enhance the global minima.Among the methods, Boltzmann method achieves the best value among.When the networks get larger or more complex, more neurons are applied in the program, Boltzmann method also have better performance than hyperbolic method and the Wan Abdullah method.It is because when using Boltzmann machine, after running for a long enough time at a certain temperature, the probability of a global state of the network will depend only upon that global state's energy, according to a Boltzmann distribution.When temperature decreases from a high temperature to reach a thermal equilibrium at a low temperature, we are guaranteed to converge to a distribution where the energy level fluctuated around the global minimum.The heat cause the neurons unstuck from local minima.As a result, Boltzmann machine is having the best result among the others methods.

Hyperb
From the overall comparison in logic programming that perform higher order Horn clause, each method had achieved low and non-ideal global minima.However for Boltzmann machine, it produces better global minima among them.

Conclusion
From the theory and experimental result, the ability of Boltzmann machine in doing logic programming on Hopfield network is better than Wan Abdullah method, which is based on Mc Culloch Pitts updating rule, and hyperbolic tangent activation function.It provides a better result in term of global minima ratio and hamming distance for higher order clauses.However, for higher order, the result of global minima obtained was very low due to the complexity of network.

Figure 1 .
Figure 1.Discrete hopfield network Figure 2 sh are weight should be