Troubleshooting
How to prove an IPC slowdown is hardware not software
Short answer
A slow HMI on a booting IPC is not fault evidence. Capture per-process CPU, memory, disk and network under the production workload first: saturation with a clean event log is a sizing or software problem, and only an error that repeats under a separate one-component stress load — twice, with a matching Windows event — convicts the hardware.
On this page
- Is a slow HMI proof the IPC hardware is bad?
- What do I capture first on the IPC?
- Which Windows records separate hardware faults from software faults?
- How do I check the drive and memory before any stress test?
- How do I stress test the IPC without destroying the evidence?
- What result finally proves the IPC hardware is at fault?
- When should I stop testing and replace the IPC?
- Questions asked
Run the production HMI workload first, not a stress test: capture per-process CPU, memory, disk and network in Resource Monitor while the slowdown happens, and judge it against fixed limits — processor saturated above 85 percent held, available memory below 500 MB, disk active time above 90 percent with a queue above twice the logical processors. High utilization with a clean event log is a sizing or software problem; only an error that repeats under a separate one-component stress load — processor, then memory, then storage — is evidence the IPC hardware itself is failing. This page gives the capture steps, the pass limits, a worked example and the verdict for each case.
Is a slow HMI proof the IPC hardware is bad? #
No — a computer that boots, displays and runs the HMI slowly has passed the startup path, and slow response with high utilization almost always means application demand, competing software or undersized hardware rather than a failed component.
Define the term before you test: a hardware fault means a component cannot perform its rated function under the specified workload and environment. High utilization caused by inadequate capacity is a sizing problem, and no stress test repairs it. The pass standard you test against is the HMI platform vendor's documented system requirements for your project, plus the IPC manufacturer's stated operating limits.
| What you observe | What it usually is | Next measurement |
|---|---|---|
| High utilization only with the HMI running | Application demand, project configuration or insufficient capacity | Per-process CPU, memory, disk and network |
| Errors during an independent hardware load | Hardware, driver, cooling or power path | Event records, temperature, drive health, repeatability |
| Normal resource use with poor HMI response | Blocked communications, graphics or security software | HMI diagnostics, network behavior, process waits |
| Boot failure, stop error or corrupted display | Direct fault evidence | Windows and device diagnostics |
What do I capture first on the IPC? #
Run the production HMI workload exactly as it runs in service, and capture total and per-process processor utilization, available memory, disk activity and network activity from Resource Monitor while marking the operator action that triggers the slowdown.
- Boot with the normal production services and security controls in place — do not "clean" the machine first, or the capture proves nothing about the state you operate in.
- Select Start, enter resmon, and open Resource Monitor with its CPU, Memory, Disk and Network views.
- Launch the HMI project and reproduce the operating state tied to the complaint.
- At the moment of the slowdown, record total processor percentage, available memory, disk active time and queue, network utilization, and the top process on each resource.
- Repeat the triggering action once with nonessential third-party software closed, within site security policy, and note what changed.
Read the capture against fixed limits, not impressions: the processor is saturated when utilization holds above 85 percent and the backlog of runnable threads stays longer than the logical-processor count; memory is pressured when available memory sits below 500 MB or committed use exceeds physical; storage is a bottleneck when disk active time holds above 90 percent with a queue above twice the logical processors. Name the process owning the saturated resource — the HMI project, the historian, antivirus, an update or an agent — because the fix follows the name, and when the hog is an edge agent the same capture method is the one used for troubleshooting high CPU usage in a Node-RED agent.
Which Windows records separate hardware faults from software faults? #
Read Reliability Monitor and Event Viewer around each reproduced slowdown: a hardware, storage or driver event that precedes the stall is fault evidence, while a clean record during reproduced slowness points back at the application, its communications or the machine's capacity.
In Reliability Monitor look at the daily summary of crashes and critical events, then open Event Viewer for the detail: application crashes, unexpected shutdowns, storage or file-system errors, corrected hardware reports, driver resets, and service failures.
Sequence decides the direction. A storage or driver reset that precedes the stall is fault evidence worth chasing; an HMI or database error followed by high resource use usually means the application is the cause and the numbers are only the consequence. Align the operating-system record with the HMI logs by timestamp before you conclude anything.
How do I check the drive and memory before any stress test? #
Check drive health and free space first, because HMI logging and antivirus share one device: read the drive's health reporting with the maker's diagnostic, keep free space above 15 percent, and run a boot-time memory diagnostic when crashes or corruption make memory a credible suspect.
Storage degrades before it dies: an SSD worn down by years of history logging shows degraded write performance while still passing a quick health glance, so treat any reported wear or error as a finding in its own right, and watch write performance fall away as free space shrinks toward that line. A memory diagnostic error is direct fault evidence; a clean memory result says nothing about storage, cooling or power, so do not let it close the case. When the write rate itself is the question, the arithmetic is the one used for choosing edge storage for high-rate vibration data.
How do I stress test the IPC without destroying the evidence? #
Load one component at a time — processor, then memory, then storage — with a stress utility from a trusted source, and stop the moment temperature, power or device-health readings leave the manufacturer's permitted range; a combined load only proves instability, never which part caused it.
Run stress loads in a maintenance window on a machine you are allowed to stop, never during production.
Pass criteria for each isolated load:
- Processor load: clocks stay in their rated range, no driver resets or corrected-error reports appear, and temperature holds within the manufacturer's limit.
- Memory load: no allocation or calculation errors reported by the utility.
- Storage load: no device or file-system errors, and latency returns to baseline once the load stops.
Only when all three pass separately, apply one combined load at the duty the machine actually faces in service. Record every run — the verdict rule below depends on what the events show, not on whether the machine merely survived.
What result finally proves the IPC hardware is at fault? #
An error that repeats under an isolated one-component load on two separate boots, with a matching hardware, storage or driver event in the Windows record, is hardware fault evidence — everything else you can capture is a sizing or software finding.
Worked example (illustrative inputs, arithmetic shown). An IPC with 4 logical processors and 8 GB RAM stalls every morning during history writes. At the stall the capture reads: total processor 42 percent, available memory 3.1 GB, disk active time 96 percent, disk queue 9.
Checks: 42 percent sits under the 85 percent saturation line and 3.1 GB is far above the 500 MB floor, so processor and memory are cleared. The storage limit is 2 × 4 = 8 queued requests, and the observed 9 exceeds it. The per-process view shows the logging service writing 40 MB/s in 4 KB records: 40 × 1,024 = 40,960 KB/s, and 40,960 ÷ 4 = 10,240 write requests per second. Against a drive rated at 8,000 sustained 4 KB random writes per second (illustrative datasheet figure), demand runs 10,240 ÷ 8,000 = 1.28, or 28 percent over rating. Verdict: the storage subsystem is undersized for the logging duty — a sizing finding, not a failed part — and a clean isolated storage stress run would confirm the drive itself is healthy.
The decision for each case:
- Repeatable error under an isolated load, reproduced from two cold boots, with a matching Windows hardware, storage or driver event: hardware fault — take both runs and the event IDs to the vendor claim.
- Saturation with a clean event record: sizing or software — add capacity or move the offending process's work.
- Normal resources with a stalled HMI: stay in the application, network and security-software path; the IPC is not the suspect.
When should I stop testing and replace the IPC? #
When the capture shows the machine running below the HMI platform's documented requirements, or the duty exceeding what the installed configuration sustains, stop testing: no stress test repairs an undersized computer, and every further run only delays the correct fix.
Put the sizing verdict on paper — processor, memory, storage type and capacity, and concurrent load against the platform's documented system requirements — so the conclusion is a recorded comparison, not an opinion. When the verdict is sizing rather than fault, the straightforward fix is replacing the box with a fanless mini PC sized above those requirements. For a host that runs around the clock, pair that sizing with the duty-cycle practices in running a compact industrial PC as an always-on server.
Questions asked
How long should each isolated stress load run?
At least 30 minutes per component, and longer when the complaint is tied to a window: if the HMI stalls two hours into a shift, carry the load across that window. A single pass proves nothing about an intermittent fault, so treat any error as unconfirmed until it repeats on a second run from a cold boot.
Would a spare identical IPC settle it faster?
Often, yes. Move the drive or software image to a known-good identical unit and run the production workload: a fault that follows the original unit is hardware, and one that follows the image is software or configuration. That swap settles the argument faster than any stress utility and gives the vendor a clean claim either way.
Can a clean stress test clear the IPC completely?
No. A clean run clears only the component you loaded, for the duration you loaded it. It says nothing about the HMI project's demands, the network, the machine's capacity, or an intermittent fault that needs days to surface, so the production-workload capture and event record stay the primary evidence.
What if CPU and memory look normal but the HMI still stalls?
Then the wait is not in the machine's core resources. Check communication timeouts, database or tag-server delays, display-driver behavior, and security software scanning the HMI's files, and use the event sequence to see which came first — a network timeout preceding the stall keeps the investigation in the software path.
Does high CPU from antivirus count against the IPC?
No. Antivirus load is software contention and one of the most common false hardware verdicts on plant-floor Windows machines. Capture the per-process numbers to prove which process owns the processor, then reschedule scans outside production hours or add documented exclusions under your security policy.