When upgrading to Ethernet-based communications, we were on the lookout for a relatively reliable and standardized solution that supported recognized redundancy standards. For mission-critical, time-sensitive substation applications, network interruptions as short as a few milliseconds can have a severe impact on system operations and jeopardize the safety of onsite personnel. For this reason, we needed a solution that could quickly connect our legacy devices to networks guaranteed to provide bumpless operation, even in the face of single points of failure. The IEC 61850 is an international standard defining communications protocols for intelligent devices at substations.

New IEC 61850 Substations

Engineers given the task of designing a new power substation have the luxury to start from scratch. When comparing conventional hardwired solutions with modern IEC 61850 solutions, many power companies are opting for the IEC 61850 solution, which can provide the same performance and reliability as a hardwired solution, but with the added benefit of scalability. Let’s look in detail at how to handle three of the main challenges engineers face when building a new IEC 61850 substation from the ground up:

• Determinism:  Will the performance of an IEC 61850 Ethernet-based substation compare favorably with that of a hardwired peer-to-peer substation?
• Device, Network, and Data Reliability:  Will the devices be able to operate reliably in environments with high EMI and how to implement network redundancy to ensure that critical packets are reliably transmitted?
• Manageability:  How can substation system engineers optimize, perform daily maintenance on, and troubleshoot an established system?

The Key to Deterministic Operation: Accurate Time Stamping for Time Synchronization.

Accurate time synchronization is required in a substation to ensure that measuring devices connected to the grid have accurate clocks. Accuracy of the clocks is measured relative to a national standard and can vary from the order of milliseconds to microseconds, depending on the application.

A network switch should identify data with highest priority and transmit those first. Quality of Service (QoS) is the common approach to transmit data based on priority tags marked on the IP packets. However, the standard QoS does not distinguish between GOOSE and SMV packets—the crucial data that require real-time transmission—and other data. Moxa has tailored the QoS function for IEC 61850 communication, which is available in our switches.

IEC 61850 QoS

In IEC 61850 substation communication, GOOSE and SMV are 2 critical packet types that require high-priority attention. To guarantee that these messages are not corrupted, they are transmitted with the highest priority, regardless of what other messages are queued up in the network. When an IEC 61850 queueing scheme is used, the Ethernet switch knows that GOOSE and SMV packets are critical, and hence always gives these messages the top priority in the sending queue.

 

The Key to Deterministic Operation:

Prioritization of Critical Data

One way to assure critical data gets first priority is to manage the network bandwidth so that data is only sent to the requestor device and not broadcasted to the entire network. The typical approaches are QoS, VLAN, and IGMP snooping. VLANs provide substation networks with the following benefits:

Trunk Traffic Reduction

Traffic can be restricted to specific network domains by assigning network devices to specific VLANs, and in this way remove potential trunk traffic bottlenecks.

Traffic Filtering

A network device assigned to a particular VLAN filters out packets sent from devices not on the same VLAN. This simple, yet effective filtering strategy is used to segregate traffic flow throughout the network.

Device Performance Improvement

Endpoint devices on the network will only process packets from the VLAN that they belong to, greatly reducing the number of messages each device needs to process.

IGMP Snooping

GOOSE and SMV are multicast packets; multicast packets in the same LAN can cause unnecessary load on host devices by processing packets they have not solicited. With IGMP Snooping, the GOOSE and SMV packets will only be forwarded to the subscribers, minimizing the bandwidth wastage.

 

Device Reliability

Devices in a substation are constantly subject to extreme temperatures, high electricity interference, and airflow restrictions. Hence, it is critical to ensure that the devices can operate reliably in harsh operating environments.

Basic Reliability: IEC 61850-3 Compliance

The IEC 61850-3 and IEEE 1613 standards precisely define EMC and communication requirements for network equipment used in substations. Substation computers and Ethernet switches must have IEC 61850-3 and IEEE 1613 certifications to guarantee adequate protection against a variety of environmental conditions. These minimum requirements include:

• Level-4 EMC for strong protection against electrical interference
• High tolerances for constant vibrations and shocks

Wire Speed Zero Packet Loss Technology: NoiseGuard™

In order to ensure no data packets (GOOSE/SMV/PTP) are lost in EMI-intensive environments, Moxa’s switches are designed with NoiseGuard™ technology that can guarantee zero packet loss in most networks. NoiseGuard™ technology is an EMC-immunity technology that exceeds the requirements of IEEE 1613 Class 2 and uses an optimized mechanical design with integrated housing for better conduction. Customized components can include a fiber transceiver, and an enhanced, optimized power circuit design.

OS Recovery: Smart Recovery™ Technology

Stable system operation is important for ensuring reliable substation management. However, when the system is unstable, how do you restore the system to normal operation as quickly as  possible? Engineers are often stymied by a lack of domain knowledge and unfamiliarity with what could be a long list of complicated recovery steps. Moxa’s computers come with Smart Recovery™, which is an automated BIOS-level software recovery tool that allows engineers to automatically trigger OS recovery to minimize downtime.

Fully Automated Recovery Tool

Special design for unmanned or remote substations that can automate the recovery process.

Two-step Manual Recovery Process

Supports an easy 2-step recovery process either from the tool or using a USB storage device, which helps the engineer to quickly

Withstand Harsh Environments: Fanless Thermal Design

Extreme temperatures ranging from -40 to 75°C, dust, and restricted airflow are conditions commonly found in substation environments that can greatly affect the performance of substation computers. Computers that rely on fans to keep their CPU cool are particularly vulnerable to these kinds of conditions. A substation computer should be fully sealed from the outside environment  and not require any type of fan. Not requiring a fan extends a computer’s life significantly, provided the computer is able to survive in the extreme heat often experienced in substation  environments. Engineers must therefore work to situate the PCB’s highest thermal concentration in the very center of the device, so that heat has the largest immediate area available to dissipate into. With fanless systems, generally the entire outer shell is utilized as one large heat sink, with fin heights, gaps, thicknesses, and points of contact carefully analyzed and adjusted to further optimize dissipation. What this means is that designing a fully fanless computer is a nontrivial engineering challenge, with fanless computers inevitably more expensive than fan-cooled solutions. But the additional cost is more than justified by the huge increase in reliability, as well as the additional benefits of reduced size, complexity, and protection against dust, heat, and corrosion.

 

Network Reliability

Packet losses of any kind are not tolerated in substation communication. Ensuring that critical packets are reliably transmitted is a key for any substation, so it is critical to implement network
redundancy to prevent data loss when failure occurs.

Network Redundancy Technology

Network redundancy is a method for ensuring network availability in case a network device or path is unavailable. There are many existing mechanisms that can enhance fault-tolerance in an Ethernet network. The most common ones are RSTP, proprietary ring redundancy, and PRP/HSR. However, in today’s complex substation automation industry, even millisecond long network interruptions cannot be tolerated because they may severely impact system operations or jeopardize on-site safety. IEC 61850 edition 2 clearly states that the transmission of GOOSE and SMV packets in substation automation systems are required to be bumpless. Moxa also provides PRP/HSR technology for zero packet loss network redundancy technology to ensuring on-site safety and quality of service.

 

Using Native PRP/HSR Computers for Efficient Network Management

Another concern of substation engineers is that if they use PRP/HSR technology, they will have to deal with multiple protocols (MMS, PRP/HSR supervision frame, SNMP) in their system, thereby increasing the complexity of management.

A native PRP/HSR computer is the best choice when it comes to ensuring high reliability while monitoring a PRP/HSR network with built-in PRP/HSR management middleware that supports both SNMP and MMS interfaces, Various substation devices that run different communication protocols, including PRP/HSR devices, can be connected to this computer. The supervision frame from the PRP/HSR devices is converted to SNMP or MMS format at the device level and then sent to the middleware for analysis. The integration of the middleware and PSCADA system enables
all data to be effortlessly used and read in the substation PSCADA system via the MMS protocol. Substation operators find it easy to manage all devices on the PRP/HSR system via the PSCADA visual tools. In addition, troubleshooting can be easy since any single-point-of-failure can be shown on the PSCADA system, making the PRP/HSR application more reliable and stable.

Enhancing Cybersecurity in Substations

In an age where cyberattacks on public infrastructure are increasing, protection of substation networks and assets cannot be taken lightly. To protect your mission critical networks, Moxa provides a portfolio of Gigabit secure routers that provide secure remote access to field devices through public networks and facilitate layered defense-in-depth network security for substation networks.

Data Reliability

Critical data such as control signals and warnings are sent in GOOSE messages from the PSCADA to the designated IEDs. A built-in mechanism in switches to detect delay, loss, or tampering of
GOOSE packets can facilitate quick troubleshooting in substation networks.

GOOSE Check and GOOSE Lock Technologies for Packet Monitoring

GOOSE Check is a type of packet inspection where the status of each GOOSE packet passing through the Ethernet switches is continuously monitored. Substation operators can then use MMS to send this information to a PSCADA system to visualize dropped, delayed, out-of-sequence, and tampered GOOSE packets, and trigger system alarms when the status of a packet changes. GOOSE Lock creates a whitelist of legitimate GOOSE packets passing through the Ethernet switches in a network. It is a protection mechanism against malicious packet-based attacks from unknown sources.

 

Manageability

How can substation system engineers optimize, perform daily maintenance on, and troubleshoot an established system?

Configuration Wizard With Less Than 7 Steps

Configuring network devices correctly can be a major headache for substation engineers, particularly since incorrect configurations can result in an unstable or nonfunctional communication infrastructure. That’s where Moxa’s proprietary Substation Configuration Wizard can make all the difference. Because substations are such a specialized environment, IT teams will only require a few key features. For this reason, it makes a lot of sense to simplify and streamline the configuration process. Reducing the configuration interface to only the relevant network features, makes setup and maintenance much more efficient. As is illustrated in the accompanying graphic, engineers can use Moxa’s browser-based configuration wizard to deploy our network devices in as few as 7 steps.

Built-in MMS Server for PSCADA Integration

Upgrading multiple peer-to-peer communication connections on the same network, and finding communication errors can be an extremely complicated exercise. With fully integrated MMS support based on IEC 61850-90-4 modeling, substation SIs and automation engineers can display their entire network of automation devices right alongside process-level information, all in a single SCADA view, which makes managing the Ethernet switch the same as managing IEC-61850 electrical devices. Therefore, you will no longer need to install and configure separate NMS software for IT devices on your substation system.

Administrators can use MMS to:

• Monitor and control IEDs, switches, embedded computers, device servers, and process data from a single power SCADA interface
• Eliminate redundant SNMP systems for IT hardware while decreasing network congestion
• Configure devices for event triggers, polling reports, or both
• Precisely locate devices relative to other devices within the network hierarchy in a single software view
• Directly configure and control IT hardware from the SCADA system

 

Sometimes substation networks deploy switches that do not have integrated IEC 61850 MMS, making network management more complex as these switches cannot be integrated easily into the PSCADA system. Moxa’s network management software supports RESTful APIs and web widgets that enable easy integration of such devices with third-party applications (e.g., web-based SCADA).

 

Proactive Diagnosis of ST/SC/SFP Fiber Issues: FiberCheck™ Technology

How can a mechanism that anticipates when fiber components are reaching the end of their life cycle be implemented, allowing engineers to replace the components before they fail?

Optical fiber, which is used in substation networks to protect Ethernet connections from electromagnetic interference, may deteriorate after long years of usage. FiberCheck™ Technology uses Digital Diagnostic Monitoring technology to diagnose deterioration of signals before the signals are too weak and drop off, offering early error detection and fault isolation for predictive maintenance. Most substations currently support only SFP-type optical-fiber monitoring.

Moxa FiberCheck™ can be used by substation switches to monitor ST/SC (as well as SFP) connectors, and notify the PSCADA system via SNMP or MMS when abnormalities are detected, allowing operators to quickly initiate maintenance procedures. Reports and alarms can be transmitted using any one of the following methods: a network port, a serial console, CLI, MMS reporting, SNMP traps, digital relay, entries written to the system log file. The FiberCheck™ function also allows system operators to monitor transmission and reception power, temperature, and voltage/current along optical-fiber connections in real time.

Performance and Protection

A powerful and secure management platform is crucial to substation automation. However, with a variety of applications operating on a single system, overall performance can experience significant degradation. What substation engineers would like to avoid is the need to maintain multiple management platforms, which can be costly and difficult to implement and maintain.

Protocol conversion is only one of the functions sought while choosing embedded computers for a retrofit substation project. Most of the time, embedded computers in a substation are used to run many different applications and operating systems.

In such cases, engineers would prefer to use virtualization technology such as VMware to run independent virtual machines, with the following benefits:

Reduced Costs

VMs increase the efficiency and utilization level of your existing x86 hardware platform, thereby saving the cost of acquiring new hardware.

Application Isolation

Depending on the capability of your hardware platform, you can run each application on a separate VM for complete isolation of the applications. You can also run critical and non-critical application workloads on separate VMs to ensure that if one set of applications fails, the other applications can continue to run.

Extend the Life of Your Legacy Applications

You can use VMs to run your legacy applications on computers with new hardware platforms or operating systems. However, don’t assume that every computer platform works well with VMware. Be sure to use products that display the VMware-ready logo, which indicates that a product meets the criteria for VMware integration and interoperability.

Monitoring the Status of Key Components for Advance Alerts: Proactive Monitoring

How do you improve the performance of your industrial computers and get advance alerts on the key performance indicators to help reduce unexpected equipment downtime? Proactive  monitoring, Moxa’s innovative software visualization tool for industrial computers, monitors the computer’s health by keeping an eye on CPU usage, memory usage, storage partition usage, the operating temperature of the CPU and motherboard, and the redundant power monitor, and can trigger relay outputs to provide either visual or audio alarms. What’s more, you can configure the tool to trigger these alarms based on user-defined criteria.

Legacy-to-hybrid Network Deployment

Substation retrofit projects require integration of existing RSTP ring networks with PRP/HSR architecture to bring DANs (dual attached nodes) to a bumpless redundant communication backbone and to enhance system availability. With Moxa’s IEC 61850-compliant devices specifically calibrated for PRP and HSR redundancy, seamless control and monitoring can be achieved, to help customers construct or retrofit their network infrastructures as required for time-sensitive and mission-critical applications.