CPU improvements + step() API

[LucasJSch]

Overview

This PR modified the Python client and server such that it improves CPU usage when using python scripts and python server.

Step API

The step() API is useful for testing here because it turns the sim into a client-driven, synchronous workload you can drive, measure, and assert on, instead of fighting real-time clock jitter and pub/sub noise.

Client fix

1. Lazy recv thread on client.py

Before the fix, connect() always started a background thread that ran this loop:

while self.state:
    frame_packed = self.socket_topics.recv(block=False)  # returns immediately if no data

When nothing was on the wire, recv(block=False) raised TryAgain and the loop restarted immediately. That is a busy-wait: one Python thread pegged a full CPU core (~100%+) even when the client was idle.

After these fixes, we only start the thread on the first subscribe() call. The benefits are:
a. Zero recv-thread CPU when you only use RPC
b. Pay the CPU cost only for pub/sub
c. The thread exists only when there is something to receive. That matches how the client is actually used.

2. Blocking recv + 100ms timeout

Before: non-blocking recv in a tight while self.state loop → spin when idle.

After: blocking recv with a 100ms socket timeout:

    def __recv_topic_frame(self):
        try:
            # Blocking recv with timeout (set on socket in subscribe()).
            # Returns None on timeout so the loop can check self.state.
            frame_packed = self.socket_topics.recv()
        except (pynng.exceptions.TryAgain, pynng.exceptions.Timeout):
            return

Benefits:
a. With no data, the thread blocks in the kernel instead of spinning. Idle wakeups drop from ~millions/sec to ~10/sec (100ms timeout).
b. Blocking recv returns as soon as a message is ready.
c. disconnect() sets self.state = False then join()s the thread. The 100ms timeout guarantees the loop re-checks self.state at least every 100ms, so join() finishes within ~100ms instead of hanging forever on a blocking recv with no data.
d. Daemon threads avoid join hangs by being killed at process exit, but that skips cleanup. Non-daemon + 100ms timeout gives orderly shutdown and a bounded wait.

How these two work together:

flowchart TD
    A[connect] --> B{subscribe called?}
    B -->|No| C[No recv thread — 0% recv CPU]
    B -->|Yes| D[Start recv thread]
    D --> E{Data on socket?}
    E -->|Yes| F[Process message immediately]
    E -->|No| G[Block up to 100ms in kernel]
    G --> H{self.state still True?}
    H -->|Yes| E
    H -->|No| I[join completes within ~100ms]

Loading

Server fix

What was wrong before: The old server recv loop looked like this:

nng_recv(..., NNG_FLAG_ALLOC | NNG_FLAG_NONBLOCK);
if (rv == NNG_EAGAIN) {
    std::this_thread::sleep_for(std::chrono::milliseconds(1));
    continue;
}

When no message was waiting:

1. NNG_FLAG_NONBLOCK made nng_recv return immediately with NNG_EAGAIN
2. The loop slept 1 ms and tried again
3. That repeated ~1000 times per second while idle
   Unlike the Python client (which had no sleep and burned a full core at ~100%), the server already had a 1 ms backoff — so baseline system CPU was only 3–6%, not catastrophic. But it was still a pointless poll loop: wake, check socket, sleep, repeat, even when nothing was happening.

1. Set NNG_OPT_RECVTIMEO (100ms) at startup

  // Recv timeout so RecvLoop blocks instead of polling with NNG_FLAG_NONBLOCK.
  // 100ms is short enough for responsive shutdown (state_ check) while
  // eliminating the 1ms poll wakeup overhead.
  const int recv_timeout = 100;
  rv = nng_setopt_ms(topic_socket_, NNG_OPT_RECVTIMEO, recv_timeout);

Benefit: Tells the socket “block up to 100ms waiting for data.” The kernel handles the wait instead of userspace polling. Same 100ms choice as the Python client — consistent behavior on both ends of the pair socket.

Also enables: Bounded shutdown. RecvLoop runs in recv_thread_, which is join()ed on teardown. Without a recv timeout, a blocking nng_recv with no incoming data could hang join() indefinitely.

2. Remove NNG_FLAG_NONBLOCK from nng_recv()

Before: NNG_FLAG_ALLOC | NNG_FLAG_NONBLOCK
After: NNG_FLAG_ALLOC only (blocking recv, subject to the 100ms timeout)

Benefit: nng_recv now waits for data instead of instantly returning NNG_EAGAIN. That is what makes the socket timeout meaningful — nonblocking recv would ignore the blocking semantics and force you back into a manual poll loop.

3. 3. Remove sleep_for(milliseconds(1))

That sleep was only there because of nonblocking recv:

nonblock recv → EAGAIN → sleep 1ms → retry

Once recv blocks with a timeout, the sleep is redundant — and harmful, because it added an extra 1 ms delay on every empty poll cycle.

Benefit: Eliminates ~1000 artificial wakeups/sec. Idle polling drops to ~10 wakeups/sec (one per 100ms timeout).

4. Handle NNG_ETIMEDOUT same as NNG_EAGAIN (continue)

    if (rv != 0) {
      if (rv == NNG_ETIMEDOUT) {
        continue;  // recv timeout — re-check state_ and retry

Before: NNG_EAGAIN → continue (after 1ms sleep)
After: NNG_ETIMEDOUT → continue (no sleep)

Benefit: A timeout is not an error — it just means “no data yet.” The loop goes back to while (state_.load()), so:

• Shutdown works: setting state_ = false is picked up within ~100ms
• No spurious warnings: only real errors hit the LogWarning + break path

This is the direct replacement for the old EAGAIN branch; the error code name changes because blocking recv with a timeout returns ETIMEDOUT instead of EAGAIN.

How this was tested

1. Build project
2. Run Blocks UE project
3. Execute PYTHONPATH=client/python/projectairsim/src python3 -m pytest tests/test_cpu_usage.py -v -s.

Results:

ests/test_cpu_usage.py::TestCPUProfile::test_case2_connected_only 
======================================================================
  CASE 2: Client connected only
======================================================================
  Python process CPU:   0.0%
  System CPU (total):   14.3%
  Python threads:       1

PASSED
tests/test_cpu_usage.py::TestCPUProfile::test_case3_scene_loaded_idle 
======================================================================
  CASE 3: Scene loaded, idle (no stepping)
======================================================================
  Python process CPU:   1.4%
  System CPU (total):   14.5%
  Python threads:       2

PASSED
tests/test_cpu_usage.py::TestCPUProfile::test_case4_step_3ms 
======================================================================
  CASE 4: Step only, dt=3ms
======================================================================
  Python process CPU:   8.6%
  System CPU (total):   14.3%
  Python threads:       2
  Steps:                1666
  Elapsed:              5.00s
  Steps/sec:            333.2
  Avg step latency:     3.00ms

PASSED
tests/test_cpu_usage.py::TestCPUProfile::test_case7_step_10ms 
======================================================================
  CASE 7: Step only, dt=10ms
======================================================================
  Python process CPU:   3.6%
  System CPU (total):   12.6%
  Python threads:       2
  Steps:                417
  Elapsed:              5.00s
  Steps/sec:            83.3
  Avg step latency:     12.00ms

PASSED
tests/test_cpu_usage.py::TestCPUProfile::test_case8_step_20ms 
======================================================================
  CASE 8: Step only, dt=20ms
======================================================================
  Python process CPU:   2.6%
  System CPU (total):   12.8%
  Python threads:       2
  Steps:                239
  Elapsed:              5.02s
  Steps/sec:            47.6
  Avg step latency:     21.00ms

PASSED
tests/test_cpu_usage.py::TestCPUProfile::test_case9_idle_after_stepping 
======================================================================
  CASE 9: Idle after stepping
======================================================================
  Python process CPU:   1.4%
  System CPU (total):   11.3%
  Python threads:       2

PASSED
tests/test_cpu_usage.py::TestCPUProfile::test_summary 
======================================================================
  PROFILING SUMMARY
======================================================================

  Case                                       Py CPU   Sys CPU  Threads
  ---------------------------------------- -------- --------- --------
  2. Connected only                           0.0%    14.3%       1
  3. Scene loaded, idle                       1.4%    14.5%       2
  9. Idle after stepping                      1.4%    11.3%       2

  Case                                       Py CPU   Steps/s   Lat ms
  ---------------------------------------- -------- --------- --------
  4. Step 3ms, no images                      8.6%    333.2    3.00
  7. Step 10ms, no images                     3.6%     83.3   12.00
  8. Step 20ms, no images                     2.6%     47.6   21.00

  Hottest thread (2_connected): TID 584563 at 0.0%

  Hottest thread (3_scene_idle): TID 584628 at 0.2%

PASSED

Credits

Co-authored-by: James Wiles (@Zaffer)