The energy sector is facing rapid changes in the transition towards clean renewable sources with power grids still forming the backbone of energy transmission as well as distribution.
However, the growing share of volatile, fluctuating renewable generation such as wind or solar energy and the increasing share of cross-regional energy exchange already led to a massive increase in grid congestion and network security concerns in recent years.
Operators are facing a complex and dynamic environment with many uncertainties. This makes maintaining grid security very challenging. If congestions occur, they are usually mitigated by either modifying generation or demand - redispatching, curtailment and flexible loads.
These necessary measures unfortunately require a deviation from the most economical use of available resources and lead to higher emissions and excessive costs in congestion management.
Congestions are the result of the complex interplay of a large number of grid elements. Individual line loads can not be simply viewed in isolation and even parallel flows resulting from transmissions in distant regions need to be considered.
Although being complex, this interdependence also has the positive implication that control settings, such as switch state and tap settings not only change the power flow in the vicinity, but affect line loads in the entire grid forming the basis for non-costly grid optimization measures.
The optimal orchestration of just a few grid actions can significantly lower the load of over-utilized lines, improve the reliability of the grid and mitigate the need for costly traditional congestion management measures.
Network topology optimization, a prime example of a non-costly congestion management measure, involves the control of hundreds of circuit breakers, giving rise to an exponential number of possible switching states. While all these combinations bear a massive optimization potential, identifying the best possible combination of actions results in an astronomical search space.
Mathematical optimization would be the method of choice to address the optimal power flow problem, but is unfortunately not suitable to handle a combinatorial search space including all readily available non-costly measures.
Except for the highly nonlinear dynamics of the grid, a very similar combinatorial problem arises from any demanding strategic board game: Planning of potential action sequences through a vast decision space.
AI-systems such as AlphaGo, AlphaZero and AlphaStar outperform world champions in competitive, strategic board games as well as rich strategic online games. These systems all have in common that they are powered by agents capable of learning from interaction with a simulated environment.
At the beginning, the agents start out with completely random (and at that point still useless) behavior. However, by alternating experience collection and strategy (behavior) improvement they gradually enhance their skill towards a given goal function throughout the course of the training eventually mastering the targeted control setting. This process is called Reinforcement Learning (RL).
A key ingredient for applying RL at scale is the availability of efficient and accurate simulations. Fortunately, this prerequisite is in general already fulfilled for electrical power networks.
The applicability of the powerful AI-based optimization method reinforcement learning to power grid control has been successfully demonstrated in international research competitions.
The operation of interconnected networks involves a multi-step planning process and close cross-regional coordination. Near-real time adoption of operational plans is necessary to accommodate for changing external influences while adhering to numerous constraints on several network elements such as voltage limits, thermal thresholds, ramping limits, phase angles, etc. Additionally, extremely high availability is demanded and unexpected outages are not allowed to induce violations of operational constraints. This is ensured by additional security constraints for all contingencies.
We demonstrated that our RL-based methodology copes with multi-period planning under uncertainty, security constraints, realistic grid dynamics in AC-formulation and grid control at circuit breaker level.
Active network topology control and optimization are readily available, fast and economic measures helping to prevent or alleviate grid congestions. This leads to improved operational security and reduces the dependency on costly measures such as redispatching or curtailment.
Our RL-based methodology makes it possible to consider topological corrections such as line switching or bus splitting in day-ahead and intraday planning. Busbar-systems can be fully customized and restricted according to individual operational constraints.
Remedial actions lists are predefined operational responses to certain grid states, e.g. after contingency events or at the onset of upcoming congestions, with the goal to keep the grid within given operational constraints. The creation of remedial action lists is an expensive, manual process, conducted by experienced grid operators which is hard to align with additional optimization goals.
Our RL-based agents assist the curation of remedial action lists. This includes simplifying the creation and maintenance of these lists, enabling thorough analysis of their effects as well as effectively targeting additional optimization goals.
Maintenance work is carefully planned to not interfere with the safe operation of the grid as part of the operational routine. While considering safety, optimality with respect to dispatch efficiency is usually given less consideration, as this presents an even harder problem.
Our RL-based optimization methodology enables safe and efficient maintenance planning.
