IMD SolutionComputing LibraryShop main store

Troubleshooting

Why PI Network Manager fails before the SAN disk is ready

Operating systems on the plant floor Published 6 min read

Short answer

PI Network Manager fails because it starts before the SAN-backed E: volume with the PI home directory is mounted. The iSCSI initiator dependency orders startup but does not prove the volume is ready; make the target reconnect and the mount persistent, add delayed automatic startup as the buffer, and verify at the next controlled reboot. The post-failover data delay is unrelated to the dependency change.

On this page
  1. What the event sequence actually shows
  2. Why the iSCSI dependency orders startup but proves nothing about E:
  3. Make storage readiness persistent first
  4. Delayed automatic startup is the buffer, not the proof
  5. Timing the race at a 03:00 weekly reboot on the primary node
  6. Apply and verify at the next controlled reboot
  7. The failover data delay is a different fault
  8. Questions asked

PI Network Manager fails on the primary node because Windows starts the service before the SAN-backed E: volume holding the PI home directory is mounted, and the buffering and interface services then fail behind it. Adding a dependency on the Microsoft iSCSI Initiator service is reasonable but incomplete: a running initiator proves a service state, not a mounted volume. The fix that holds is making storage readiness itself persistent, with the target session restored at boot and the volume mounted under its drive letter, plus delayed automatic startup as the timing buffer, verified at the next controlled reboot. The data-update delay that appeared after the interfaces failed over to the primary is a different fault; service dependencies act only at start and stop.

What the event sequence actually shows #

The event log already contains the diagnosis. The Network Manager failure event lands before the disk-arrival event, which means the application start ran ahead of usable storage, so the fix has to change that order or that timing, not the PI configuration. Read the failure, disk-arrival and volume-ready timestamps as one sequence:

What you observed What it establishes Decisive check
Network Manager fails, then the SAN disk connects Application start precedes usable storage Compare the failure and disk-arrival timestamps, not just their order
Buffering and interface services fail too They inherited the missing prerequisite Read each service's own failure event and configured dependencies
The secondary node reboots clean Its storage path or boot timing differs Compare storage presentation, service paths and startup types before copying settings
Data updates lag after failover to the primary A separate path, not the dependency change Correlate interface and SDK message logs with arrival times

An interface service that will not start has failure modes of its own beyond this race; the PI-to-PI interface service will not start from FactoryTalk Historian case covers the post-install variant, so read that service's event separately before folding it into the boot problem.

Why the iSCSI dependency orders startup but proves nothing about E: #

Windows service dependencies are ordering controls: the dependent service waits until the prerequisite reaches the running state, and Windows stops the dependent service first at shutdown. Between the initiator running and the PI home directory being readable sit several more state transitions, and the dependency covers only the first:

  1. The network path to the storage system is up.
  2. The Microsoft iSCSI Initiator service is running.
  3. A session to the target is logged in.
  4. The disk is presented to Windows.
  5. The volume mounts with drive letter E:.
  6. The PI home directory is readable by the service account.

The dependency you added ends at step 2; Network Manager needs step 6. If the initiator reports running while the session is still logging in or the volume is still mounting, the race survives the dependency unchanged. A mounted volume and a readable directory are also different tests: verify the service account's access to the path itself. One operational effect belongs in the change record: the dependency reverses shutdown order, so Windows now stops Network Manager before the initiator, which matters on a node in a failover pair.

Make storage readiness persistent first #

Work the connection with the server and storage team, not the PI side:

  • Confirm E: actually rides the Microsoft iSCSI Initiator. SAN storage can reach Windows by other transports, and the dependency must match the real connection.
  • Configure the initiator so the target session is restored automatically at boot, using the reconnect-at-startup option on the configured target, so reconnection depends on neither a login session nor manual action.
  • Confirm the volume mounts without interaction and keeps its drive letter; a disk that arrives without E: mapped still fails the service.
  • If the sessions ride a specific NIC, treat any change there as its own project: migrating an iSCSI initiator to a new NIC without data loss follows the same session-by-session readiness chain.

With the session and the mount persistent, the dependency does its real job of ordering, and the startup delay becomes margin instead of the load-bearing fix.

Delayed automatic startup is the buffer, not the proof #

Setting the PI services to delayed automatic startup defers their launch past the initial automatic-services wave; the deployment in question uses a two-minute offset, and that delay is adjustable. Elapsed time is not a storage-readiness test. It is a buffer you size against the measured volume-ready time, so the durable arrangement is persistent reconnection to make the disk arrive early, the dependency to make the service wait its turn, and the delay to absorb the weeks when the SAN is slow.

Timing the race at a 03:00 weekly reboot on the primary node #

Inputs, each read from the node's own System event log at the weekly reboot; the clock values below are example readings in the format the log gives, to be replaced with yours:

  • Reboot completes at 03:00:00 (the maintenance window).
  • Microsoft iSCSI Initiator service reaches running at 03:00:35.
  • The E: volume reports ready at 03:02:35.
  • Delayed automatic startup defers the PI services by 2 min = 120 s, the configured and adjustable offset.

Rule in words: the buffer works only if the delayed-start moment falls at or after the volume-ready moment, with room for the SAN's week-to-week spread.

Arithmetic:

  • With the dependency, Network Manager may start once the initiator runs: 03:00:35, which is 35 s after boot.
  • Volume ready: 03:02:35 − 03:00:00 = 2 min 35 s = 155 s after boot.
  • Dependency-only start: 155 s − 35 s = 120 s too early.
  • Delayed start at 120 s: 155 s − 120 s = 35 s too early. The two-minute buffer still loses this race by 35 s.
  • Delay needed to clear this reading: more than 155 s, so at least 3 min before any margin is added.

Decision: on these timings the buffer alone fails. Keep the dependency for ordering, make the session and mount persistent so the 155 s collapses toward the initiator's 35 s, and hold the delay as margin. The threshold to act on after every reboot: if volume-ready minus boot exceeds the configured delay, here 155 s > 120 s, extend the delay or fix the storage path rather than rebooting again to see what happens.

Apply and verify at the next controlled reboot #

  1. Record for Network Manager, and for the buffering and interface services, the startup type, service account, executable path and dependency list.
  2. Compare those against the secondary node and note the real differences in storage presentation and startup type; do not copy its settings wholesale.
  3. When adding the iSCSI prerequisite, append it to the existing dependency list rather than replacing the list, and confirm the prerequisite's exact identity in Windows service properties instead of assuming its short name.
  4. Check for circular dependencies and walk the expected start and stop order before the reboot.
  5. Reboot inside a controlled window with a known failover state and an agreed data-continuity check, capturing the three timestamps (initiator running, volume ready, Network Manager start) and the final state of every service.

The failover data delay is a different fault #

The dependency change cannot slow data while the services run: dependencies act at start and stop only. The delay appeared when the PI-to-PI interfaces failed over to the primary, so work the failover path, meaning interface reconnection, source retrieval, buffering and destination delivery, correlated through the interface message logs and SDK logs against event timestamps. When values arrive current but history trails behind, the archive and trend side is the place to look; the diagnosis for PI showing live values but no trend history walks that split. And if the weekly reboot exists to clear an unresolved problem, fixing that root cause removes the race from your calendar entirely; until then, the storage fix above is what makes each reboot safe.

Questions asked

Does the iSCSI initiator dependency slow down PI-to-PI data updates?

No. Service dependencies act only when services start and stop; they do not sit in the data path while the services run. The delay that appeared with the failover comes from the failover path itself, meaning interface reconnection, buffering and delivery, and it is diagnosed in the interface message logs and SDK logs against event timestamps, not by reverting the dependency.

Is delayed automatic startup enough on its own?

Only while the measured volume-ready time stays under the configured delay. With a two-minute delay, any reboot where E: passes 120 seconds after boot still fails the service. Size the delay from your own timestamps, volume-ready minus boot plus margin for the SAN's slowest week, and treat a breach of that threshold as a storage problem rather than a reason to keep adding delay.

Why does the secondary node start cleanly when the primary does not?

Because its storage path or boot timing differs, not because PI is configured better. Compare the two nodes' storage presentation, drive mapping, service paths, dependencies and startup types, then close the specific gap you find. Copying the secondary node's service settings without that comparison can break a node whose storage arrives differently.

What should we capture at the next controlled reboot to prove the fix?

Three timestamps from the System event log: initiator service running, E: volume ready, and Network Manager start, plus the final state of the buffering and interface services. If volume-ready minus boot exceeds the configured delay, the buffer lost and the storage path needs work. If the service starts after the volume with margin, the fix held for that reboot.

Should we keep the weekly reboot at all?

If the reboot works around an unresolved fault, fixing that root cause removes the boot race from your calendar entirely. If the reboot is policy, keep it, but verify storage readiness and service start at every one, because a silently failed Network Manager on the failover node is an outage waiting for the next failover.

Keep reading

  • How-to

    Configure IP networking on an edge PC with no SSH access

    A new edge PC that leaves the factory with DHCP and static addressing disabled has no IP address, so SSH cannot connect to it. Do the first configuration at a keyboard and monitor: set X1 to DHCP and the plant-side port to the planned static address, restart networking, then confirm both addresses with nmcli device status and ip a before opening SSH or browser management on port 9090.

    Operating systems on the plant floor

  • How-to

    Identify the Linux base before installing Node-RED natively

    Run cat /etc/os-release and uname -m on the running controller before copying anything. ID=debian opens a Debian package workflow; ID=ptxdist with PTXDIST_VERSION 2020.08.0 marks an embedded image where an ARMv7l Node.js tarball needs the hard-float libatomic1_13.2.0-4_armhf.deb staged by the image's own method. The product label never decides this.

    Operating systems on the plant floor

  • Troubleshooting

    When AOA Manager shows server stopped after install

    Server Stopped in AOA Manager is a service state, not an install state: open services.msc and confirm AOANginx, AOANodeRed and AOAPython are running. A standalone Node-RED install proves nothing about them. On Windows 7 Embedded 32-bit, AOA 2.6 is outside its documented requirements, which start at 64-bit Windows 10 or 11 Professional and 16 GB of RAM.

    Operating systems on the plant floor