macOS host support for otbr-agent (RCP mode) — working port, seeking guidance before PRs

[clara-mostly-human]

---

Summary

I have otbr-agent running as a full border router on macOS (Apple Silicon
Mac mini, bare metal, RCP mode with a SONOFF EFR32MG24 dongle). Real
Matter-over-Thread devices (two Nuki smart locks) are commissioned through
it into Home Assistant and work reliably, including across host reboots.

The surprising part: the tree is nearly there already. A stock checkout
gets 96% through compilation with -DOTBR_MDNS=mDNSResponder. It took two
small changes in this repo plus one in openthread to reach a fully working
border router:

1. src/rest/json.cpp: a C++ VLA fails Apple clang with -Werror
   (-Wvla-cxx-extension); fixed with a fixed-size buffer.
2. src/mdns/CMakeLists.txt: don't link a standalone dns_sd library on
   Apple platforms (the API is in libSystem).
3. openthread src/posix/platform/infra_if.cpp: IPV6_BOUND_IF takes an
   interface index on macOS, not a name; the shared BSD branch passed the
   name and died with EINVAL at startup. (Separate PR to
   openthread/openthread.)

What works (verified with real hardware)

• RCP over HDLC/UART (460800 baud, EFR32MG24)
• utun-based Thread netif; network forms, state=leader, dataset persistence
• mDNS via native mDNSResponder: _meshcop._udp advertised; discovered by
  Home Assistant on the LAN
• SRP server + advertising proxy: Thread devices' _matter._tcp records
  published to the LAN with AAAA resolution
• Border routing: RA emission with PIO + RIO on the infra interface works
  out of the box (tcpdump-verified); LAN hosts learn routes and reach mesh
  devices; Matter CASE sessions established host->mesh
• Full Matter commissioning of Thread devices via HA + phone flow

Platform requirements found

• sysctl net.inet6.ip6.forwarding=1 is required (macOS defaults to 0).
  Without it the host neither forwards infra<->utun traffic nor delivers
  packets arriving on the infra interface for utun-owned addresses.
  Question for maintainers: should otbr-agent set this at startup on macOS
  (matching what the Linux setup scripts do), or should it be a documented
  deployment requirement?

Not attempted / out of scope for a first PR

• Backbone router: Darwin has no usable IPv6 multicast routing (MRT6
  equivalent); would need a userspace forwarder. Single-BR Matter homes
  don't need it.
• Firewall: macOS pf could host the equivalent of the ip6tables rules;
  proposed as a follow-up PR.
• NCP mode (the existing netif_unix.cpp stub already documents this).

Questions before I open PRs

1. Is macOS host support something the project wants to carry? (I'm happy
   to also contribute a macOS build job for GitHub Actions to keep it
   green.)
2. PR structure preference: one PR with the two build fixes + CI job +
   docs, or split?
3. The forwarding sysctl question above.
4. For the openthread infra_if.cpp fix I'll open a separate PR there —
   anything else platform-related you'd want bundled with it?

Code

Branches with the changes described above:

• https://github.com/clara-mostly-human/ot-br-posix/tree/macos-support (this repo: 2 commits)
• https://github.com/clara-mostly-human/openthread/tree/macos-posix-fixes-main (openthread: 1 commit, rebased on main)

Happy to share logs, packet captures, or run experiments — the setup is my
daily-driver home automation so it gets continuous real-world testing.

[jwhui]

@clara-mostly-human , thanks for submitting this issue!

Is macOS host support something the project wants to carry? (I'm happy to also contribute a macOS build job for GitHub Actions to keep it green.)

It would be great to add macOS support!

PR structure preference: one PR with the two build fixes + CI job + docs, or split?

We generally prefer to break separable changes up into separate PRs.

The forwarding sysctl question above.

Following the existing Linux implementation works for me.

For the openthread infra_if.cpp fix I'll open a separate PR there — anything else platform-related you'd want bundled with it?

In general, anything needed to get full functionality on macOS implemented and testsed is appreciated.