Comparison of Different Auction Pricing Rules in the Electricity Market

The objective of this study is to investigate the impact of different auction pricing rules on the market performance in the context of the competitive electricity market. In pursuing this objective, a simplified model of auction based electricity market has been designed and three distinctive pricing rules are analyzed: uniform pricing, pay-as-bid pricing and the Vickrey-Clarke-Grove pricing. Using agent-based modeling approach, generators have been modelled as agents submitting price-quantity bids to the market. The Simulated-Annealing Q-learning algorithm has been adopted as the learning mechanism for the agents so they can maximize their profit using strategic bidding. The computer simulation is used to test the effect of the different pricing rules on the total dispatch cost, bid price and generators’ profit. In addition, the generating capacity of one of the competing agents is altered to a significantly larger size to evaluate the effect of the relative market share on total dispatch cost and agents’ welfare. This study concludes that the pay-as-bid auction can complicate the way bidders learn and react about each other’s strategy. While uniform pricing results in high and volatile total dispatch cost in the market and pay-as-bid pricing induces truthful bidding leading to low dispatch cost, Vickrey pricing seems to provide a good balance between controlling the total cost and its stability. The theoretical and practical contributions of this study are also discussed in the paper.


Introduction
The electricity industries reform around the world is aimed at improving systems efficiency, attracting investments in the market, as well as ensuring a competitive electricity price.The advent of deregulation, privatization and market competition has created a new paradigm in the electricity industry through the management and modeling of an auction based electricity market.While competition exists in both the seller side and the buyer side of the market, competition in the supply side mainly has become the driving factor for generation companies to employ more efficient plants and generation technologies.Competition in electricity trading also highlights the importance of an effective market mechanism to ensure fair and contestable market exist for generating companies and new players.However, defining policies and market rules are not easy because of the complex interactions among market participants, which are mainly profit driven.In fact, the selection of the pricing rule has been a major issue in the electricity market establishment process.
In the earlier stage of the market establishment, Uniform auction has been the widely adopted pricing rule in the USA, the European and the Australian markets.In uniform auction, winners are paid at a single market clearing price for their supply instead of their bidding price.Proponents for Uniform rule believe that it induces truth telling yielding a market that is efficient and easy to implement.However, Uniform auctions have been perceived to cause poor performance in some deregulated market (Fabra, 2002), price spikes (Kahn, Cramton, Porter, & Tabors, 2001).The alternative to the Uniform pricing rule is the discriminatory (or pay-as-bid) pricing rule.In the pay-as-bid rule, winners are paid according to their bids.Xiong, Okuma and Fujita (2004) studied the implementation of this rule in the UK market and reported that the pay-as-bid pricing rule resulted in a reduction of price volatility and lower average prices.Despite the favourable arguments for the pay-as-bid pricing rule, for example in (Federico & Rahman, 2003;Fabra, Fehr, & Harbord, 2006), some skeptical views remain, stating that the pay-as-bid rule will result in inefficiency in a highly competitive market and induce significant entry barriers and penalizes smaller generators (Vazquez, Rivier, & Perez-Arriaga, 2001).
While the debate pertaining to the competitive electricity market has been focused on comparing the advantages and disadvantages of the uniform and the pay-as-bid pricing rules, little attention has been paid on Vickrey auction rule.Such is the current situation in the literature despite Vickrey auction being highly recommended by economists.Vickrey rule is originally formulated for a single-unit auction.It is a type of second-price-sealed-bid auction.When the highest bidder wins, the price paid is the second-highest bid.Vickrey claims that this mechanism gives bidders an incentive to bid their true value.For the multi-unit electricity market auction, a modified version of Vickrey pricing rule called Vickrey-Clarke-Groves or VCG (Clarke, 1971(Clarke, , 1973) ) has been introduced.The mechanism of VCG is that each bidder in the auction pays the opportunity cost that their presence introduces to all the other bidders.The intention here is that, by making the payment received by a bidder independent of its own offer price, marginal cost bidding can be induced as a weakly dominant strategy.In other words, this modified version of Vickrey pricing rule maintains the incentive to bid truthfully.Despite the extensive study of Vickrey rule in the economics discipline, it has received relatively little attention in the electricity market studies (Fabra, Fehr, & Harbord, 2006).
Given the essence of competitive market settlement is characterized by its pricing mechanism, there have been much research efforts on studying the implication on pricing rules in settling the electricity market.While some studies are based on theoretical analysis and mathematical derivation (Zhang, Luh, & Wang, 2000;Federico & Rahman, 2003;Cramton, 2004;Hinz, 2004), more recent approach employs agent based simulation model.The shift in the employed methodology is because: (1) the analytical studies ignore the repetitive nature of the trading (Rothkopf, 1999) that in fact leads to the emergence of important phenomena in electricity market trading, such as the implicit collusive behavior identified by Axelrod (1981), and (2) given that the electricity market is a complex system, agent-based model is deemed suitable as the simulation approach to understand the dynamics of a competitive market and the phenomena that emerge as a result of repetitive interaction among market players.
The use of agent based model to conduct electricity market analysis has been increasingly popular.Using this approach, generators bidding to the market are coded as agents that interact with the market environment.Artificial intelligence can be embedded in agents to make them autonomous and adaptive.Popular AI techniques, such as Neural Network, Genetic Algorithm, Game Theory, and Reinforcement Learning can be implemented as agents' mechanism to make strategic bidding.Veselka et al. (2002) defined the agents' learning strategy based on Genetic Algorithm.Bunn andOliveira (2001, 2003) introduced an agent-based model to mimic the interactions between the power exchange and balancing mechanism.The agent's learning policy is based on a strategic stochastic game, with a utility function to facilitate the exploration-exploitation mechanism.Other studies employing agent based models include: Amin (2002) to observe the evolution of the electricity industry, Watanabe, Yamaguchi, Shiina and Kurihara (2004) to study stochastic unit commitment, and Tellidou and Bakirtzis (2007) to study capacity withholding and tacit collusion.
In this study, an agent based model has been developed and implemented to conduct computerized experiments to examine the market behavior and bidding strategy when different pricing mechanism is employed to resolve the auction market.Q-learning algorithm has been adopted to model the learning behavior of generator agents when strategically bidding into the competitive market.This Reinforcement Learning algorithm accounts for the repetitive trading in the auction market and allows generator agents to remember and learn from their bidding experience.The objective of the study is to evaluate the impact of employing different pricing mechanism on the market performance and overall welfare of the competing generators.Total dispatch cost announced by the independent system operator (ISO) in resolving the auction is chosen as the indicator for market performance, as the ISO always resolve the auction based on the lowest possible dispatch cost.In this study, total dispatch cost is the sum of the revenue received by all generators.Generators' profit is a reliable indicator reflecting the welfare of the competing generators.This study explores whether or not the Revenue Equivalence Theorem holds in a multi-unit multi-period oligopoly market.According to this theorem: in a symmetric auction environment when all players are risk neutral, different pricing rules yield the same total dispatch cost.Given that the clearing price calculation is associated with auction type and it has certain influence on generator's bidding behavior, we hypothesize that RET does not hold in a multi-unit multi-period market.In this paper, the term "pricing rule" is used to represent "auction type" because the study focuses on the determination of market clearing price according to the auction type in study.The study makes a contribution to existing auction theory by contrasting the effect of different auction types in an oligopoly market context.The study generates a promising and more realistic insight compared to conventional economic analysis given that most economic analysis does not model the heterogeneity nature of the bidders and assumes continuous supply function.The simplification and assumption made by the analytical approach fail to capture the micro level dynamics in the model.A bottom up modeling and simulation approach has a strategic advantage in that it enables observation of possible emergence at the macro level, such as price spike, generation mix and emission patterns.The research findings has practical implication by extending current understanding on the pros and cons of uniform and pay-as-bid pricing rules, as well as introducing the potential benefit of the VCG pricing rule.
The rest of the paper is structured as follows.Section 2 presents Complex Systems as the paradigm to perceive the competitive electricity market problem.It also outlines the motivation for adopting the Simulated Annealing (SA)-Q-learning algorithm to model the intelligence for the generator agents.Section 3 presents the design of the agent-based simulation used in this study.It provides descriptions of dispatch cost and generator's profit calculation under different pricing rules.It also includes mathematical model depicting the action-selection policy of the generator agent and its learning mechanism.Section 4 explains the experimental design of our study, outlining the parameter values used in the computer simulations.It also reports the simulation results and discusses the research findings in detail.Section 5 summarizes our key findings and extends them to relate to the current literature.This concluding section also highlights the theoretical and practical implications of this study.

Agent's Learning Algorithm
The scheduling-pricing-dispatch problem in electricity market is unique and complex.While the electricity trading related to the pricing aspect of the problem is governed by the rule of supply and demand, the scheduling and dispatch aspects follow electrical engineering principles which are non-linear in nature.Agent based simulation model has gained popularity, especially for studying complex systems.The novelty of using an agent based model to study complex phenomena is that artificial intelligence, such as learning ability, can be embedded to build agents' cognitive and decision making ability.Agents' responsive and purposeful behavior in algorithmic form can be encoded as computer programs.To embed intelligence in our proposed agent-based model, we surveyed the literature in search for the most suitable reinforcement learning technique.
Reinforcement learning introduced by Sutton and Barto (1998) is an algorithm for agent to take actions in an environment with the aim of maximizing an anticipated long-term reward.Instead of acquiring knowledge by learning from examples, in reinforcement learning, agent acquires knowledge by learning from experience.The algorithm attempts to find a policy that maps states of the world to the actions the agent ought to take in those states through the assignment of positive and/or negative rewards.Q-learning is a specific type of reinforcement learning that assigns values to action-state pairs.Agents' learning is guided by an action-value function that assigns an expected utility value of an action in a particular state.The algorithm populates a Q-table as a mechanism for an agent to remember and learn from its past experience.Cell values are updated as agents explore the effectiveness of taking actions in certain states.When values in Q-table have been sufficiently updated, agents are equipped with sufficient knowledge to make strategic move, such as to make strategic bidding to maximize generators' profit.In other words, the agent has sufficiently learnt the action-value function to the extent that it has acquired some knowledge to define the optimal policy for selecting the best action leading to highest utility value.The advantage of the Q-Learning algorithm is that it can readily compare the expected utility of available actions without modeling the complex environment.This feature makes it suitable for decision making problems in repeated games with unknown components (Xiong, Okuma & Fujita, 2004).
In addition, there are two types of actions for an agent to undertake, namely exploration action and exploitation action.In exploration action, an agent is to explore unknown possibilities selecting the action-selection policy.In exploitation action, an agent utilizes its best known action to undertake strategic action leading to a rewarding experience.There are several exploration and exploitation combinations to define learning policies -the most popular type is the -greedy approach.As the name suggest, -greedy chooses (or exploits) action associated with the highest estimated reward.In the exploration mode, an action is selected at random following the uniform distribution.This random sampling is to ensure that optimal actions have been discovered when sufficient sampling trials have been conducted.It should be noted that -greedy approach is effective for static search space.Given that a sufficient number of search attempts are allowed, a strategic bid as an optimal solution can most definitely be found in a static setting.However, the nature of the electricity trading is such that the market environment is (1) dynamic -changing from one trading context to another, and (2) exhibits non-linear search space, due to parameters such as transmission constraint, ramp rate and discrete supply function.Hence, applying -greedy policy will not necessarily lead to an optimal solution.In a modified version of Q-learning, called Simulated Annealing (SA) Q-learning, the balance between exploration and exploitation actions can be refined by means of a temperature dropping factor (as per Simulated Annealing method).

The Proposed Agent-Based Simulation Model
Figure 1 illustrates the proposed simulation model with Q-learning coded as the learning algorithm for the generator agents.In the competitive electricity market, generator agents submit bids to the independent system operator (ISO) in each trading period.The ISO, represented as the Market Environment in Figure 1, is responsible for clearing the market in each trading period based on a predetermined pricing rule.
The bid price that a generator agent offers ranges from its production cost and the price cap set by the ISO, whereas the bid quantity ranges from the minimum stable load to the maximum generation capacity.This is shown in Equations ( 1) and ( 2). (1) (2) Where Cost g is the production cost of a competing generator agent g; and are the bid price and bid quantity submitted by a competing generator agent g at trading period t; is the price cap set by the ISO; and is the minimum stable load and maximum quantity that can be offered by a competing generator agent g.
Dispatch optimization is performed using bid stacking method with bids are ranked from lowest bid price to the highest.The ISO schedules generators' dispatch up to the level that satisfies demand at any trading period t: where G is the total number of generators.
Generators, which compete for dispatch, always adopt a profit maximizing strategy.The Q-Learning algorithm is aiming to maximize the overall reward the generator agent can obtain during the simulation.In this study, the profit a generator agent receives in a trading period t is defined as the immediate reward .When the Uniform pricing rule is adopted to govern the bidding process, all the generator agents with non-zero dispatch will be paid at a single market clearing price.Therefore, a generator agent g's profit at trading period t is calculated as: (4) where is generator agent g's profit at trading period t; is the market clearing price at trading period t, which is set by the highest bid price of all generator agents with non-zero dispatch; is actual dispatch quantity scheduled to the generator agent g at trading period t.
When the pay-as-bid pricing rule is adopted to govern the bidding process, each generator agent will be paid based on its own bid price.Therefore, a generator agent g's profit at trading period t is calculated as: where is the generator agent g's bid price at trading period t.
When the Vickrey pricing rule is adopted to govern the bidding process, a generator agent will be paid at the opportunity cost that its presence introduces to all the other generator agents.Therefore, a generator agent g's profit at trading period t is calculated as: where is the total utility received by all generator agents, i.e. the total cost incurs to the ISO, when generator agent g is competing for dispatch at trading period t; is the total utility received by all generator agents at trading period t when generator agent g is excluded from the competition; is the total utility received by all the other generator agents when generator agent g is competing for dispatch; ev is the revenue received by generator agent g at trading period t.

Defining the Action
An action can be described as a strategic move made by an agent in each simulation step.In this study, when an agent takes an action, it is making a bidding decision, which is defined by the price-quantity bid that a generator agent submits to the ISO for the next trading period.For each generator agent, there are N main intervals; and within each interval, there are M sub-intervals defined.N is the number of quantity selections that can be chosen by a generator agent with each selection represents a bid quantity range; M is the number of price selections that can be chosen by a generator agent with each selection represents a bid price range.Hence, taking an action is to locate the corresponding interval according to generator agent's price and quantity bid.The values for N and M are calculated as follow: In a situation where a generator agent g is allowed to bid at its own preference, equals to ; when this generator is required to bid at its maximum capacity, is set to -1.In the latter case, there is only one bid quantity available for selection as N equals to 1.With this design, we can simulate scenarios for maximum quantity bidding or variable quantity bidding.Consequently, the impact of variable quantity bidding can be analyzed.It should be noted that many existing studies often assumes bidding at maximum quantity in electricity market simulation.

Defining the State
A state describes an agent's current condition prior to its next movement.In this study, it is defined as the price-quantity bid a generator agent submitted in the previous trading period.In other words, there is a one-to-one relationship between each action taken in the previous trading period and each state following the trading period.Hence, it is the state is modeled in the same manner as the action: there are N times M sub-intervals in total representing all possible states.

Defining the Action Selection Policy
In this study, the Simulated Annealing (SA)-Q-Learning algorithm is adopted as the action selection policy for generator agent.As proposed by Guo, Liu and Malec (2004), the SA-Q-Learning was designed to balance the exploration and exploitation during the learning process and has proven its performance in such activities.The SA-Q-Learning algorithm defines the action selection process as follows: The tempe Assuming number, is a consta algorithm.

Ramp R
The bid qu where of the gene

Learni
The learni value for t study, the Thus, repe environme been execu informatio Hence, the where

Fine-T
Computer sufficient n to a stable 70,000 for the first 10 iterations.

Discount Factor
In the Q-Learning algorithm, the discount factor models the agent's preference of a recent reward versus a long term reward.In real life context, defining long term within the repetitive electricity market trading is rather obscure because there is no clear indication for a terminating point.Actual dispatch information for each generator is updated at each trading day.Thus, a generator needs to adjust its bidding strategy dynamically according to its bidding performance in the previous trading days to compete effectively with the other generators.It is therefore reasonable to assume that the generator agents prefer most recent reward to guide their learning.In this study, the discount factor is assigned a value of 0.1 based on trial and error.This value allows a short reaction time for the generator agents to respond to the change in the market.

Case Studies
A three-by-three-by-two experimental design has been setup to explore the effect of the pricing rule on the market performance.The pricing rules trialed in the simulation are: (1) Uniform, (2) Pay-as-bid (or Discriminatory) and (3) Vickrey.In each case, three scenarios are tested.In the three scenarios, the maximum generating capacity of generator agent 1 is set to: 25% of the aggregated supply capacity, 50% of the aggregated supply capacity, and 75% of the aggregated supply capacity.These three scenarios are set to simulate situations where: (1) generator agent 1 has no dominant position, (2) generator agent 1 has a dominant position but cannot fulfill demand by itself, and (3) generator agent 1 can fulfill demand by itself, accordingly.Each case includes four competing agents with similar attributes, namely the same minimum stable load, ramp rate and production cost.The aggregated (or total) supply capacity of the four generators is also fixed.Such setting allows us to focus the experiment and observe solely on the impact of the pricing rule or the supply capacity.
Furthermore, each scenario is tested for two variations, which are, maximum quantity bidding and variable quantity bidding.In the first variation, the agents must submit bids that offer their maximum generating capacity; while in the second variation, generator agents can submit bid quantities following their profit maximizing strategies.This means, in the second scenario, generator agents may offer less than their maximum available capacities when they consider this as strategic bids.
According to the state-action model presented previously, the main technical difference between these two variations is the dramatically increased number of state-action pairs under the variable quantity bidding.The huge increase in the number of explored state-action pairs indicate more learning iterations required in the simulation process.As an illustration, there is approximately 1500 available actions to choose for agent 1 (with 75% of the aggregated supply capacity) in each iteration.
The remarkable increase in the number of iterations performed can be illustrated by the time taken to run the simulation.Using a computer with 4GB DDR3 memory and a quad-core Intel core i7 mobile processor, each simulation of maximum quantity bidding took two to three days to complete.With the second variation whereby the agent is not forced to bid at its maximum capacity, each simulation run takes up to three weeks to complete.
In terms of the number of iterations in each simulation case, the uniform pricing simulation is stopped after 300,000 iterations, while the other two cases are stopped after 500,000 iterations.Each simulation run has been repeated 20 times and observed patterns are consistent in these repeated runs.

Uniform Pricing
Three scenarios are simulated with uniform pricing employed as the rule to determine the market clearing price.When the market share is equal, generator agents tend to take risk by bidding at high price to increase their overall profit.Figure 3(a) shows the bid price of a generator agent in the last 2000 iterations in the simulation.As can be seen in Figure 3(a), the agent tends to bid at the production cost with frequent price spikes indicating bidding at a much higher price than the production cost.The resulting profit earned by the generator agent can be seen in Figure 3(b).It is also noted that the bidding behavior of the other agents follow the same fashionexhibiting risk taking and gaming behavior.This bidding behavior results in high volatility in the total dispatch cost and the market clearing price.
When a generator agent has an increased market share, collusion among the other agents is observed.Figure 4 (a) displays the bidding prices of generator agent with larger market share, while Figures 4 (b), (c) and (d) show the bidding prices of the other generator agents.Although the three agents with smaller generating capacity do not communicate to each other, they seem to conspire against generator agent 1 with larger generating capacity.

Discuss
The objec performan e also es in some trading periods, it effectively changes the supply-to-demand ratio in those trading periods.As a result, a slight difference in the total dispatch costs can be observed in Figure 9(b) (referring to the bar chart when the x-axis is labelled 25%).
When the market share is asymmetric, we conclude that the theorem does not hold.Evidently, the pay-as-bid pricing rule yields the lowest cost.It is observed that the volatility in the total dispatch cost is reduced significantly because generator agents tend to bid truthfully according to their private cost function.This is in support of the experimental finding of Rassenti, Smith and Wilson (2003) involving human bidders and the finding of Xiong, Okuma and Fujita ( 2004) with their multi-agent simulation approach.Yet, the simulation indicates that the cost and the profit resulting from the pay-as-bid are rather unstable due to the complexity and high level of uncertainty in the market environment.For example, when there is a dominant agent bidding at a very high price, the other (smaller) agents do not respond immediately by increasing their bids to take advantage of the high price.
The finding of this simulation study also aligns closely with the finding of an economic analysis conducted by Fabra, von der Fehr and Harbord (2006) concluding that uniform auction result in higher average prices (or total dispatch cost) than pay-as-bid auction.Further, we also note that the total dispatch cost with uniform pricing rule is highly sensitive to the capacity of the dominant generator agent in the market.Consequently, the cost can increase rapidly.On the contrary, the pay-as-bid pricing rule can dramatically reduce the total dispatch cost even though this cost can be unstable.
Vickrey auction is a second-price auction with some kind of interdependency relating the agents' bids.Our results indicate that agents bid in a dynamic fashion with volatility observed in the agents' bid prices and profit.Yet, the method to calculate the clearing price (and thus, the profit of the winning bidder) is quite complicated that this auction also encourages truthful bidding.It is fair to conclude that the interdependency in agents' bids provides an indirect learning mechanism for agents to speculate about each others' bidding behavior.The results of our simulation show some coalition exists in which agents seem to take turn in winning bids and thus, making profit.When maximum capacity bidding is required, the Vickrey pricing rule produces the highest total cost when the dominant generator agent can fulfill the entire demand.Nevertheless, this scenario is not common in real life.Hence, in a slightly unequal market share observed among the players, the Vickrey pricing rule seems to provide a good balance between controlling the total cost and its stability.
All in all, the findings of this study complements current studies using theoretical analysis which inform the design of auction based market.In electricity market design, researchers are interested to explore market efficiency, optimal bidding strategy and revenue comparison.This study provides insights on how different auction types can influence bidding behaviors of generators.In addition, the profits earned by the generators and resulting total dispatch cost incurred by the market operator have been compared.
A major contribution of this study is the extension to previous research on how auction rule affects the likelihood of collusive outcome.Theoretical work on this issue is scant.Using agent based simulation, we observe tacit collusion in uniform pricing auction.Agents with smaller generating capacity tend to form coalition in gaining opportunities in the presence of an agent with larger generating capacity.In situation when forming a coalition is not effective, agents attempt to inflate the market price by withholding their generation capacities.
In most of existing study, including that by Gaofeng, Hashiyama and Okuma (2002), Son, Baldrick andLee (2004), Wu et al. (2004) and Tellidou and Bakirtzis (2007), agent are forced to bid their maximum generating capacity.This study also extends the work of existing studies by investigating the effect of allowing variable quantity bidding.This scenario is more realistic, although it is very costly in terms of simulation time.In particular, agent's learning time during the exploration stage increases dramatically.When variable capacity bidding is allowed and market share is unequal, smaller agents tend to inflate the price by withholding their capacity.This withholding phenomenon is seen in uniform and Vickrey auctions when bidders refer to an interdependent value function.
The practical implication of this study is that the findings inform the policy makers and regulators in terms of the capabilities and limitations of the different pricing rules.Given that the electricity market in each region is unique in terms supply to demand ratio, market share, price transparency, not to mention the maturity of the market, it is important for the electricity market to be designed carefully to ensure that (1) it is attractive for new players, which hopefully will bring about newer and cleaner power generation technology into use, and (2) it results in a competitive electricity price which can indirectly enhance the competitiveness of other commodities in many other industries in the region.