Analysis of players behavior and electricity markets conditions through simulation can assist market decision-makers in appropriate designing and policy-making before actual implementation and can reduce costs and possible future problems. Considering ancillary service markets in addition to the energy market in simulations makes possible to compare and evaluate different market auction and commodity allocation models more efficiently. In this article, impacts of implementing different models of sequential and simultaneous allocation of energy, spinning reserve and replacement reserve on players behavior, possibility of price reversal, and payments in the markets are evaluated. In this regard, first, an approach based on the multi-agent system and the reinforcement learning algorithm is presented for simulating players behavior in the energy, spinning reserve and replacement reserve markets. Then, this approach is applied on a sample system including 17 generators, with various sizes and supply costs, under sequential and simultaneous allocation models. The analysis and evaluation results obtained about price reversal conditions and payments under different models will be useful for electricity markets policy- and decision-makers.