Troubleshooting
Fixing RAPID IPC messaging when queues go silent
Short answer
Queues found by a PC scan after a warm start are leftovers, not live connections: RAPID sends to the queue name it holds, which no longer matches the PC listener. Connect to RMQ_TROB1, create a GUID-named incoming queue, send that name to RAPID, keep one PC client, and bound every retry.
On this page
Silent IPC queues after a warm start are a naming failure, not a broken controller: the PC has connected to queues left over from the earlier session, while RAPID is sending to a queue name that no longer has a listener. Do not reuse discovered queues. After every start, connect the PC to the controller's RAPID incoming queue RMQ_TROB1, create a fresh incoming queue named with a GUID, send that name to RAPID, and let RAPID address the PC by it. Run one PC client, bound every retry, and tear the queues down on shutdown. That procedure restores message flow in exactly this situation — a PC application that scans queues, connects without errors, and then receives nothing — and it removes the retry storm that shows up as random crashes and 100% CPU.
Why the queues you find are dead #
The Robot Application Builder PC library gives you an Ipc class with GetQueue, CreateQueue and DeleteQueue, and an IpcQueue object to send and receive on. A successful GetQueue proves only that a queue object exists and can be opened; it says nothing about whether anyone is sending to it. Discovery is naming information, not a health check.
The failure looks like this, as reported for a virtual controller running two RAPID tasks, T_ROB1 and TGovernor, with one PC thread per task: after a warm start the RAPID program runs normally, T_ROB1 reports its sends as successful, and the PC receives nothing. The PC scans, finds the four queues from the earlier run, cannot delete any of them, connects without a single error — and the queues stay silent. Both sides are telling the truth: RAPID wrote to a queue that exists, and the PC is listening on a queue that nothing is addressed to.
In that same setup the PC could not reconnect to the controller's outgoing queues after a warm start, while the controller's incoming queue RMQ_TROB1 stayed reachable. That asymmetry is the whole basis of the fix: stop trying to resurrect the old outgoing path and route reconnection through the one queue that never goes away.
These observations come from an IRC5-family virtual controller with the Robot Application Builder library. Treat the handshake below as the default for any PC client that must survive warm starts, and expect the same queue lifecycle on a real controller until your own round-trip test shows otherwise.
The reconnect handshake that restores flow #
// PC-side session naming in C#; controllerQueue and pcQueueName are illustrative variables
string controllerQueue = "RMQ_TROB1"; // RAPID incoming queue in this setup
string pcQueueName = Guid.NewGuid().ToString(); // a fresh name on every start
- Connect to RMQ_TROB1 through the documented Ipc entry points; this queue stays reachable after a warm start.
- Create the PC's incoming queue under pcQueueName, a new GUID on every start. Never reuse a name from an earlier session.
- Send pcQueueName to RAPID over RMQ_TROB1 as the handshake message.
- RAPID stores the received string and addresses every message to this PC session by that name.
- Prove the path with one round-trip test message before any job traffic starts.
- On shutdown or reconnect, tear down this session's queues and references, then run the handshake again from step 1.
Use GetQueue, CreateQueue and DeleteQueue as the documentation presents them; the queue name is the only thing you generate yourself.
One PC client, no exceptions, unless the RAPID code explicitly tracks several client names and their connection states. A second client can always create another queue — that is not the same as being a supported second connection, because RAPID has to know which name to address. With both T_ROB1 and TGovernor messaging, one client holds both conversations; if you keep a thread per task, the threads share that single session and synchronize every access to it.
What to log so silence has a cause #
Log the queue name, the operation and the result on every call, in both directions. With that in place, a silent path always resolves into one row:
| What you check | Where | What it decides |
|---|---|---|
| Name RAPID currently holds | handshake traffic on RMQ_TROB1 | destination matches the GUID created this session |
| Result of every send | Ipc call return on the sending side | failed send versus silent listener |
| Queue full and not ready returns | send and receive paths | flow control the application must honor |
| Messages actually received | PC receive-loop log | end-to-end delivery, independent of discovery |
Bounded retries: the arithmetic #
The controller queue is fast but has limited capacity, and queue full and not ready returns happen from time to time; that is reported for this setup, and it is flow control, not a fault. A queue's capacity is fixed when the queue is created, which is the same reason bounded real-time queues are preallocated, not warmed; a full queue always means the reader is not draining it.
Worked example with illustrative numbers: alive polled once per second on each of two tasks (2 messages/s) and status polled twice per second on T_ROB1 (2 messages/s) gives 4 messages/s nominal. Bounded retry policy: on a failed send, retry after 50 ms, then 100 ms, 200 ms, 400 ms and 800 ms — five attempts spanning 50 + 100 + 200 + 400 + 800 = 1550 ms — then log a communications fault and stop retrying. Worst case, every one of the four sends per second fails and burns its full budget: 4 × 5 = 20 attempts/s, which the PC absorbs. A tight retry loop with no wait has no ceiling at all: it retries as fast as the CPU turns the loop, pins a core, and backfills the queue faster than the reader drains it — the reported all-CPU symptom.
Keep polling to what the control task needs: event-driven status where you can, alive at 1 Hz. Alive traffic is supervision, never safety — the controller's safety chain, not a PC queue, must handle a PC that stops talking.
If it still crashes once the queues flow #
The handshake removes the trigger, not a thread defect. After delivery is proven in both directions, a crash or a pegged core is PC-side: two threads writing the same buffers without synchronization, or a receive loop spinning on empty receives. Give each thread its own receive wait, have each loop wait between polls, and pass data between threads through one synchronized structure. The method is the same as troubleshooting high CPU usage in a Node-RED agent: log what every loop does and find the loop that never waits. The layout itself — PC code running beside controller tasks — is standard, the same as when you run Python and Node-RED beside PLC code on an edge PC; loops own their work, one session per client, synchronization at the boundaries.
Questions asked
Can I just delete the leftover queues at startup?
You can try, because the Ipc class does expose DeleteQueue, but in the setup that produced this failure no leftover queue could be deleted, so never make deletion a precondition for running. Treat scan results as read-only discovery output and build the session on a fresh GUID-named queue every start. The leftovers do no harm once RAPID addresses only the name you hand it.
How do I confirm the handshake worked before starting production?
After sending the queue name, have RAPID answer with one known test message on that queue and check the receive log shows the GUID created this session, not an older name. Then send one message from the PC to RMQ_TROB1 and confirm the RAPID log received it. Only after both directions carry traffic do you start job commands, and repeat the pair after every warm start.
What does a persistent queue full return actually mean?
It means the reader is not draining the queue, because capacity is fixed when the queue is created and cannot grow at run time. Occasional full or not ready returns are normal flow control; back off and retry within your bound. If the condition survives the whole retry budget, stop sending and inspect the receiving task, because it is stuck, busy, or no longer scheduled.
Is a thread per RAPID task the wrong design?
No. One PC client with a thread per conversation is workable once the session, the GUID queue name and every shared buffer are protected. What breaks it is two threads sharing one queue object or one buffer without synchronization, or a receive loop that spins when nothing arrives. Give each thread its own receive wait and pass data between threads through a single synchronized structure.