Introduction

The purpose of a Hot Standby system is to be ready to perform a switchover, if needed. A switchover is the immediate transfer of control of the network from the primary controller to the standby controller. The transfer needs to be swift and seamless.

The M580 Hot Standby system continuously monitors ongoing system operations, and determines if a condition requiring a switchover exists. On each scan, both the primary controller and the standby controller verify the health of the system.

The primary controller verifies the health of the following:

  • the Ethernet RIO network link

  • the Hot Standby link between the primary and standby controllers

The standby controller verifies the following:

  • the health of the primary controller

  • the identity of modules in both the primary and standby backplanes

  • application versions running in the primary and standby controllers

  • firmware versions of the primary and standby controllers

  • the health of the Hot Standby link between the primary and standby controllers

Before each MAST task, the primary controller transfers to the standby controller system, status and I/O data, including date and time data. On switchover, the standby controller applies this time data and continues the same time stamping sequence. The maximum amount of transferable Hot Standby data depends on the controller.

NOTE: Both the primary controller and the standby controller maintain independent event logs. If a switchover occurs, the events recorded in the log of the former primary controller will not be included in the event log of the new primary (formerly the standby) controller.

Switchover Causes

Any one of the following events will cause a switchover:

  • The primary controller has encountered a blocking condition and entered the HALT state.

  • The primary controller has detected an unrecoverable hardware or system error.

  • The primary controller has received a STOP command from Control Expert or the DDDT.

  • An application program is being transferred to the primary controller.

  • Primary controller power is turned off; a power cycle occurs.

  • The following events simultaneously occur:

    • The primary controller loses communication to all RIO drops.

    • The Hot Standby link is healthy.

    • The standby controller maintains communication with at least one RIO drop.

Similar to a switchover, a swap is a controlled event that transfers control of the network from the primary controller to the standby controller. A swap can be caused by:

  • Execution of the DDDT CMD_SWAP command by either program logic, or an animation table Force command.

  • Manually clicking the HSBY Swap button in the Task tab of the controller Animation window in Control Expert.

Events that Do Not Cause Switchover

These events DO NOT cause a switchover:

  • simultaneous interruption of communication with all RIO drops by both the primary and the standby controller

  • partial interruption of communication with the RIO drops by the primary controller

  • a Modbus connection break

  • overload broadcast traffic generated by a peer (for example, SCADA, or another controller)

  • a BMENOC0301/BMENOC0311 module that stops operating

  • removal of an SD memory card

  • for a Hot Standby safety system, if the primary controller is partially (either the SAFE program or the PROCESS program) in the HALT state, and not all of the tasks in the standby controller are in RUN

Switchover Execution Time

If both the primary controller and standby controller are operating normally, the Hot Standby system detects a switchover causal event within 15 ms.

For both a safety and non-safety-related controller system, the effect of the switchover on the application reaction time is:

  • 15 ms for the I/O driven by the MAST task.

  • 15 ms + TTASK for the I/O driven by the FAST or the SAFE task, where TTASK is the configured execution period for that task.

The application response time for a swap or a switchover can be calculated.

After the switchover, the former standby controller becomes the primary. In the worst case, the new primary controller operates with data of scan cycle N, while the outputs have received (from the former primary controller) data of scan cycle N+1. The new primary controller re-evaluates outputs beginning with scan N+1. As the Hot Standby switchover evaluation occurs during the MAST task, some FAST task program execution may be skipped.

Switchover Effect on Main IP Address Assignments

Distributed equipment uses the Main IP address setting, configured in the IPConfig tab, to communicate over an Ethernet network with the primary controller. On switchover, the Main IP address setting is automatically transferred from the former primary controller to the former standby – now the new primary – controller. Similarly, on switchover the Main IP address + 1 setting is automatically transferred from the former standby controller to the new standby.

In this way, the configured links between the distributed equipment and the primary controller do not need to be edited in the event of a switchover.

NOTE:

  • A switchover does not affect the assignment of IP address A or IP address B. These assignments are made exclusively by means of the A/B/Clear rotary switch on the back of the controller, and are not affected by a change in primary or standby Hot Standby status.

  • When connecting Control Expert to the Hot Standby system, use IP address A or IP address B to maintain the connection on a switchover. Avoid using the Main IP address, because on switchover this becomes Main IP address + 1 and will disconnect Control Expert.

Switchover Effect on Remote Outputs

For RIO drops, the switchover is transparent: the state of outputs is not affected by the switchover. During Hot Standby operations, each controller maintains an independent, redundant owner connection with each RIO drop. Each controller makes this connection via IP address A or IP address B, depending on the A/B/Clear rotary switch designation for its controller. When a switchover occurs, the new primary controller continues to communicate with I/O via its pre-existing redundant owner connection.

NOTE: The switchover may not be transparent with respect to distributed equipment outputs.

Switchover Effect on Communication Module State

In a high availability (Hot Standby) configuration that includes BMENOC0301/BMENOC0311/BMENOC0321(C) communication modules, set the Watch Dog of the appropriate task (MAST or FAST) to a value equal to or grater than the default setting of 250 ms. Smaller Watch Dog values may cause the communication modules to timeout and enter a non-configured (NOCONF) state.

Switchover Effect on Distributed Equipment Outputs

The behavior of distributed equipment outputs during a switchover depends on whether the equipment supports hold up time. If the device does not support hold up time, its outputs will most likely go to fallback when the connection with the primary controller is interrupted, and will recover their state after reconnecting with the new primary controller.

To achieve transparent behavior, the outputs need to support a sufficiently long hold up time.

Switchover Effect on CCOTF Changes

After the standby controller becomes the new primary, it operates using both the firmware and the application previously configured in it. If CCOTF changes were previously made to the former primary controller that were not transferred to the former standby controller, these changes are not included in the configuration running in the new primary controller.

For example, assume that an I/O module was added to a remote I/O drop in the configuration running in the former primary controller. If the changed configuration was not transferred to the former standby controller, the added module will not be included in the configuration running in the former standby controller when it becomes the primary controller after switchover.

Switchover Effect on Program Logic Changes

A logic mismatch condition exists when changes have been made to the application in the primary controller, but not to the standby controller. If the LOGIC_MISMATCH_ALLOWED flag is set, the standby controller can continue to operate as standby while a logic mismatch exists. In this case, if a switchover occurs, the new primary controller executes its own, different application using data received from the former primary controller.

Depending on the nature of the application modification, different results occur:

Modification to initial primary controller logic:

Effect on new primary controller program execution:

Only code is changed (no changes to variables).

All variable values exchanged between the controllers remain the same (EQUAL).

New variables were added.

The new variables are not used by the new primary controller.

Existing variables were deleted.

The new primary controller includes the deleted variables in program execution, and applies the most recent values to these variables.

Switchover Effects on Time Management

In an M580 Hot Standby system, the primary controller and the standby controller operate their own system timers, which are not automatically synchronized. Because both the primary controller and the standby controller share a common configuration, both can be configured to perform as NTP client or NTP server.

When the NTP client function is enabled in a Hot Standby system, the primary controller and the standby controller independently receive time settings from a designated NTP server.

When the NTP server is enabled in a Hot Standby system, only the primary controllers performs the role of server.

Before each scan, the primary controller transfers system data to the standby controller, including the following primary controller system time values:

  • time of day

  • application counters

  • free running counter

On switchover, the former standby controller – now the new primary controller – applies the system time values sent by the former primary controller. Thereafter, the new primary controller continues to execute the application in the same time context as the former primary controller. If the NTP server function is enabled for the Hot Standby system, the new primary controller begins to perform the function of NTP server.

Switchover Effects on IPsec Connections

On switchover, the former primary BMENOC0301/BMENOC0311 module closes all connections that use its main IP address. These connections are re-opened on the new primary BMENOC0301/BMENOC0311 module using the main IP address after the two modules swap their main and main+1 IP addresses. As IPsec connections take a relatively long time to establish, it can take up to 5 minutes to re-establish an IPSEC connection that uses the main IP address.

Switchover Effect on Safety Operating Mode

When an M580 safety Hot Standby controller switches from standby controller to primary controller, the operating mode is automatically set to safety mode.

NOTE: The operating mode setting of a safety Hot Standby controller – either safety mode or maintenance mode – is not included in the transfer of an application from the primary controller to the standby controller.

Recovery of Former Primary Controller

The former primary controller may or may not become the standby controller, depending on cause of switchover.

If the switchover was caused by:

Make the former primary controller the standby by:

Primary halt (non-safety-related controller)

performing an INIT command and RUN the controller

Primary halt (safety controller - Process and/or SAFE task)

performing an INIT command (Process task) and/or an INIT_SAFETY command (SAFE task), and then RUN the controller

controller stop in a non-safety-related controller, or in both the Process and SAFE tasks of a safety controller

running the controller

Primary error detected

performing a controller RESET command

Application transfer on Primary

completing the transfer and RUN the application

Primary power off

powering up the controller

Loss of all RIO drops (if any) while HSBY link is still healthy and Standby controller has access to the drops

causing the controller to recover RIO drops

DDDT command

The former primary automatically becomes the standby, provided the necessary preconditions exist, for example:

  • Firmware mismatch is allowed, if a firmware mismatch exists.

  • Logic mismatch is allowed, if a logic mismatch exists.

  • Online modifications are allowed, if modifications have been made.

Control Expert HSBY Swap button