Cascading failures and blackouts are the most significant threats for power system security. If the process of cascading failures proceeds by further line tripping, the system will face uncontrolled islanding. Establishing of uncontrolled islands with deficiency in active or reactive power balance is the main reasons for system blackouts.

Controlled islanding is an active and effective way of avoiding catastrophic wide area blackouts. It is usually considered as a constrained combinatorial optimization problem. However, the combinatorial explosion of the solution space that occurs for large power systems increases the complexity of this problem. This study proposes a controlled islanding algorithm using genetic algorithm to find a suitable islanding solution for preventing the initiation of wide area blackouts by un-damped electromechanical oscillations or voltage instability. The objective function used in this controlled islanding algorithm is the minimal power-flow disruption. In this study to improve voltage stability after separation, minimal active and reactive power disruption is considered.

 The sole constraint applied to this solution is related to generator coherency. In the first step of the algorithm, the generator nodes are grouped using normalized spectral clustering, based on their dynamic models, to produce groups of coherent generators. In the second step of the algorithm, GA search for islanding solution that provides the minimum power-flow disruption while satisfying the constraint of coherent generator groups is determined by grouping all nodes. Simulation results, obtained using MATLAB and DigSilent software on the IEEE 39 bus test systems show that the proposed algorithm is efficient and have a well performance regarding voltage and transient stability when solving the controlled islanding problem. Also, results compared with spectral clustering algorithm that is newest and strongest research in this field.