Troubleshooting
Why gRPC ingestion fails when the TCP port test still passes
Short answer
A passing TCP port test proves only that the listener accepted a connection at that instant; gRPC ingestion also needs a live service, a negotiated HTTP/2 session and capacity behind the port. When both store-and-forward nodes drop together and burst on recovery, diagnose the historian host's service lifecycle, saturation and endpoint limits, not the network.
On this page
A TCP test that connects to the historian's port proves exactly one thing: at the instant of the test, something on that host accepted a connection. gRPC ingestion needs more than that — a live application endpoint, a negotiated HTTP/2 session, and ingestion capacity behind the listener. When both Store and Forward nodes stop together, log refusals like Target machine actively refused it, and then flush a burst on recovery, the fault is on the shared historian host: service lifecycle first, then resource saturation, storage latency, and endpoint-level limits. Work it with time-correlated logs from both senders and the host, and treat the passing port test as irrelevant to cause.
What the passing TCP test actually proves #
TCP reachability and application readiness are different layers. A handshake can complete while the process behind the port is overloaded, mid-restart, rejecting new calls, or unable to allocate another stream — and the port can answer a test run a minute later even though the listener was briefly gone. Each error string in your logs carries specific, limited information:
StatusCode=Unavailableis gRPC status code 14: the service was unavailable at call time, most likely a transient condition that retrying can correct. It names the condition, never the cause — a restart, saturation, an intermediary, or a lost connection all surface as the same code.Failed to connect to endpointandError starting gRPC callmean the call never got going. The next thing to establish is whether it was refused, timed out, or died at HTTP/2 setup.Target machine actively refused itis stronger than it looks: the host itself rejected the connection attempt, so the listener was down or transitioning at that exact moment. Only timestamps catch that event; a test run afterward never will.
One trap compounds all of this: the error timestamp can lag the event. In C- and Python-based gRPC clients, a request already sent on a connection that silently drops does not fail until TCP gives up retrying — about 15 minutes later. Read the event time off the traffic graph's drop to zero, not off the log line.
Where on the historian host this comes from #
The drop-to-zero-then-burst shape is a backpressure signature: normal ingestion, complete interruption, reconnection, backlog drain. In Canary's design, the endpoint your nodes dial is the Store and Forward (SaF) service on the receiving host — the version 26 Store and Forward documentation describes the sender buffering data whenever it cannot connect to that receiving service — so its lifecycle is as much in scope as the historian core. Work these mechanisms in order:
- Service lifecycle. A restart, crash, watchdog action, or process recycle makes the listener disappear; both nodes fail near-simultaneously and reconnect once initialization completes. Establish whether each restart was planned, crash-triggered, watchdog-initiated, or operating-system maintenance.
- CPU and memory saturation. Sustained CPU delays connection handling and stream processing; memory pressure pages, fails allocations, or terminates the process. Worker-thread starvation is the classic TCP-passes-while-gRPC-fails state: the listener keeps accepting while application work stops progressing.
- Storage latency. When ingestion workers cannot drain to disk fast enough, internal queues grow until upstream calls are delayed or refused. This mechanism shows in the counters before it shows in the logs.
- Endpoint and path limits. HTTP/2 stream or connection limits at the endpoint, or at a proxy, load balancer, inspection device, or NAT layer in between, appear as failures that begin at a repeatable connection or stream count. Throttling and internal queue caps are plausible, but confirm them through counters, logs, and documented configuration before changing anything.
The capture that settles it #
- Normalize clocks on both SaF nodes, the historian host, and any intermediary, and export timestamps at sub-second precision so the first sender error can be ordered against the first host event.
- Mark one complete incident on the traffic graph: last normal record, first error on each node, the zero-throughput interval, first successful reconnect, peak recovery traffic, return to baseline.
- Compare the two nodes' first-error times — the worked example below gives the arithmetic and the threshold.
- Pull the historian's application and ingestion logs for the same window and search for startup, shutdown, unhandled exception, rejected request, queue growth, timeout, throttling, storage delay, and resource-exhaustion messages.
- Pull the host's service and event logs for process exit, restart, watchdog action, memory pressure, disk faults, updates, and reboots.
- Trend across the window: CPU, available memory, paging, storage latency and queue depth, network errors, open connections, active and failed gRPC calls, ingestion rate, and queue or backlog occupancy wherever the system exposes them.
- Capture connection behavior during the next event: handshakes refused, accepted then immediately reset, or maintained while calls fail. That split separates listener loss from application-level rejection.
- Record endpoint identity — host, port, name resolution, certificates, proxy path — from both nodes, and confirm it is identical in normal and failed periods.
Also check whether trend clients failed in the same window. Readers and writers failing together means the whole service or host was down; writers only means the ingestion path specifically. Read-side faults are a different fault set, covered in troubleshooting historian trend client connection failures.
Match the signature to the fix #
| Correlated signature | Primary action |
|---|---|
| A historian restart lines up with both nodes failing | Fix the crash, watchdog, or maintenance trigger, then verify the listener stays up through the former failure window |
| CPU or memory sits at sustained pressure before the drop | Move the competing workload off the host, fix the consumer, or add capacity sized to the measured peak |
| Storage latency and ingestion queues climb together | Fix the storage bottleneck; confirm steady-state queue occupancy actually falls rather than raising queue limits |
| Failures begin at a repeatable connection or stream count | Read the documented concurrency limits at the endpoint and every intermediary, then reduce or distribute connection pressure once the limiting layer is identified |
TCP stays established while calls return StatusCode=Unavailable |
Concentrate on application health, ingestion workers, dependencies, and intermediary HTTP/2 behavior |
| Only one node fails | That node's own service lifecycle, local resources, DNS result, endpoint configuration, and connection path |
When the host itself is the ceiling and the competing workload cannot move, replace it with a fanless mini PC sized above the measured peak ingestion load plus headroom, not the daily average.
Working one outage window: two nodes, one passing test #
Inputs, each with its source:
- Two Store and Forward nodes — the deployment in question.
- Traffic graph: throughput hits zero at 09:42:00 and the first successful reconnect lands at 09:47:00. Both timestamps are illustrative; read yours off your own graph's time axis.
- Node A's log shows its first error at 09:42:20, node B's at 09:42:31 — illustrative values, each taken from that node's service log.
- The routine TCP port test at 09:44 connects successfully.
- Documented values: in C- and Python-based gRPC clients, a request on a silently dropped connection fails about 15 minutes later, when TCP stops retrying;
StatusCode=Unavailableis status code 14; the sending Store and Forward buffers whatever it cannot deliver.
The rule, in words: a TCP accept at time T proves the listener answered at T and nothing about any other moment. First errors within the same minute on two independent nodes implicate the shared endpoint. A silently dropped call surfaces roughly 15 minutes late, so an error arriving sooner than that was a fast refusal.
The arithmetic:
- Outage length: 09:47:00 − 09:42:00 = 5 min.
- Node-to-node spread: 09:42:31 − 09:42:20 = 11 s.
- Latest an error could appear if the connection had died silently mid-call at the zero: 09:42:00 + 15 min = 09:57:00. Both observed errors landed before that, so these were fast refusals — consistent with
Target machine actively refused itduring a listener transition, not a silent hang. - Backlog at reconnect, at a steady per-node rate of r records per minute: 2 nodes × 5 min × r = 10r records arriving on top of live traffic. It scales linearly with outage length: the same two nodes at a 40-minute outage hold 2 × 40 × r = 80r records.
Result: two independent nodes failed 11 s apart, the port test passed 2 min into the outage, and the errors were fast refusals rather than delayed timeouts.
Decision: a spread under one minute on independent nodes commits you to the historian host — pull its service, application, and OS logs for 09:41 to 09:43 and trend CPU, memory, and storage latency across that window, and spend no time on the network path. Only a spread of several minutes, or one node failing alone, moves the diagnosis to that node's own service, DNS, and endpoint configuration.
Keep the recovery from causing the next drop #
Recovery traffic is live data plus everything buffered during the outage, from both nodes. If the combined arrival rate exceeds what the ingestion path sustains, queues rebuild mid-drain and the cycle repeats — the sawtooth on your traffic graph. Two disciplines hold:
- Fix the trigger before touching tuning. Retries preserve delivery through a short outage, but an aggressive reconnect loop adds load exactly while the host is recovering, and a larger queue postpones failure while raising memory demand. Change concurrency, queue, timeout, or retry settings only where the component documents the control and measurements have identified the limiting layer.
- After the fix, confirm two things through the next former-failure window: the listener stays up, and steady-state queue occupancy falls during the drain instead of climbing again.
One data-integrity rule sits alongside them: never purge a sender's buffer to stop the bursts, because that buffer is the plant's record for the outage. The same drain-after-reconnect behavior over MQTT, and how to keep the backfill orderly, is covered in MQTT store and forward backfill after a connection outage.
Questions asked
What does StatusCode=Unavailable actually mean?
It names the condition, not the cause: the service was unavailable when the call was made, most likely transient. A restart, saturation, an intermediary, or a lost connection all produce the same code. Separate them by correlating error timestamps with the historian host's own logs, never by the code alone.
Why does throughput return as a burst after every outage?
Because the sender buffers locally while it cannot reach the receiving service, then drains everything it retained once the connection returns. The historian temporarily absorbs live data plus the full backlog from both nodes, and a host already near its CPU, memory, or storage ceiling can saturate again during the drain, which is what a repeating sawtooth on the traffic graph shows.
Should we increase retries or queue sizes to ride through the drops?
No. Retries preserve delivery through a short outage, but a fast reconnect loop adds load exactly while the historian is recovering, and a bigger queue postpones failure while raising memory demand. Fix the trigger first, whether restart, saturation, or storage latency, and change concurrency, queue, timeout, or retry settings only where the component documents the control and measurements identify the limiting layer.
How fast should a failure appear after the connection dies?
A refused connection fails immediately, which is what the message Target machine actively refused it records. A connection that drops silently mid-call is worse: in C- and Python-based gRPC clients the call fails only when TCP gives up retrying, about 15 minutes later, so the logged time can lag the real event by that much. Use the traffic graph's drop to zero as the event time.
Can we trust the TCP port test at all?
Treat it as one timestamped data point. It proves the listener accepted a connection at that instant and says nothing about the minutes around it, the HTTP/2 session, or the ingestion capacity behind the port. Timestamp every test and compare it against the outage window instead of trusting a pass recorded before or after the event.