Skip to content

test_models: fuzz TX messages against panda safety - #3723

Open
ping-dev-ui wants to merge 1 commit into
commaai:masterfrom
ping-dev-ui:fuzz-tx-messages
Open

test_models: fuzz TX messages against panda safety#3723
ping-dev-ui wants to merge 1 commit into
commaai:masterfrom
ping-dev-ui:fuzz-tx-messages

Conversation

@ping-dev-ui

@ping-dev-ui ping-dev-ui commented Sep 7, 2026

Copy link
Copy Markdown

TITLE: test_models: fuzz TX messages against panda safety (#32425)

Closes commaai/openpilot#32425 (bounty).

What this adds

Two tests in test_models.py, run for every platform with a route:

test_panda_safety_tx_fuzzy replays route CAN into both the car interface and panda safety so both derive the
same vehicle state, then builds the CarControl controlsd would send for fuzzed controls and asserts every message
the car controller emits passes safety_tx_hook. A rejection is a mismatch between the limits openpilot applies to
its commands and the ones panda enforces on them.

Three things it does differently from a plain random walk:

  1. Engagement is never forced. enabled is taken from panda's own controls_allowed, derived from the replayed
    CAN exactly as on the car. The existing test_panda_safety_carstate already asserts panda's engagement agrees
    with CarState, so this test inherits that and a rejection can only be a limits mismatch, never an artifact of
    the test making the two sides agree.
  2. Scenarios aim at the boundaries. Besides random holds: hold_windup holds the actuator bound for half the
    window so the rate limiters wind fully up, then steps to zero; churn toggles engagement while a command is
    live; cadence_jitter drops control frames so the real time rate checks see an irregular cadence.
  3. Coverage is reported. Sent messages are counted per (address, bus) and included in the failure message, so
    a rejection says what was being exercised, and a silent "sent nothing" cannot pass.

test_panda_safety_tx_edge covers the direction the fuzz test cannot: it takes an accepted steering command,
decodes it, re-packs it unchanged (and asserts that is still accepted, so a rejection is never a round-trip
artifact), then re-packs it with the steering signal at the DBC field's own extreme and asserts panda rejects it.
openpilot never emits such a frame, so an accept-only fuzzer would stay green if panda stopped enforcing a cap.
Table-driven per brand (STEER_CMD_SIGNALS), with a candidate list per entry because a brand can span CAN generations
that send different messages (Hyundai: LKAS11 on CAN, LKAS/LKAS_ALT/LFA on CAN FD, chosen by flags in the
car controller); the first accepted frame whose address matches a candidate is the one probed. Brands or platforms
with no matching candidate skip.

Also: Fuzzy.real() in opendbc/testing.py, same edge strategy as integer().

A note on mutation.py, because I got this wrong in the first push and CI caught it. I had deleted the
known_survivors allowlist after a local mutation run reported 3020/3020 killed. CI then reported exactly those
three surviving. The safety suite includes fuzzy tests seeded per process, so a lucky seed can fail under any mutant
and my local "kill" was noise. Line 188 is in fact an equivalent mutant: 250001U / 2U is still 125000 under
unsigned integer division, so no test can ever distinguish it. The allowlist is restored unchanged in this PR. The
two sign-boundary mutants at 218/219 (> 0 to > 1, >= 0 to >= 1) look deterministically killable with a
probe at desired_angle_last of exactly one CAN unit; I would rather do that as a separate small PR against
common.py than widen this one.

Results

Both tests run on every platform with a route. Locally (WSL, host-side libsafety), on thirteen platforms chosen to cover
every steering type and both longitudinal modes:

platform steering tx_fuzzy (25 examples) tx_edge
TOYOTA_RAV4_TSS2_2023 angle (LTA) + torque msg pass pass, both extremes rejected
NISSAN_LEAF angle pass pass, both extremes rejected (unsigned field, negative factor, 1310 offset)
TESLA_MODEL_3 angle (VM) pass pass, both extremes rejected (field spans ±1638 deg against a 360 deg cap)
VOLKSWAGEN_ID4_MK1 curvature (MEB) pass pass, +extreme rejected, magnitude field so no -probe
HYUNDAI_SONATA torque (CAN, LKAS11) pass pass, both extremes rejected
KIA_SPORTAGE_5TH_GEN torque (CAN FD, LKAS) pass pass, both extremes rejected
GENESIS_GV80 torque (CAN FD, LKAS_ALT/LFA) pass pass, both extremes rejected
KIA_SORENTO_HEV_4TH_GEN torque (CAN FD) pass pass, both extremes rejected
SUBARU_OUTBACK torque pass pass, both extremes rejected
HONDA_CIVIC_2022 torque pass skipped, no table entry yet
FORD_F_150_MK14 angle-typed, curvature safety pass skipped, no table entry yet
PSA_PEUGEOT_208 angle skipped, dashcamOnly skipped
TESLA_MODEL_X angle, alpha long skipped, dashcamOnly skipped

Zero rejections from the fuzz test on every platform that ran it. The full test_models suite with MAX_EXAMPLES=1, as CI runs it: 2610 tests, 0 failures, 376 skipped (the pre-existing dashcamOnly, SecOC and notCar skips), 379 s on 24 workers.

No new mismatches surfaced on these ten. That is a real result, not a disappointment: I would rather report that
the current limits agree on the routes tested than tune scenarios until something fails. The scenarios and the
edge probe are the coverage that was missing; the table for tx_edge is easy to extend brand by brand.

Runtime

Slowest platform seen (TOYOTA_RAV4_TSS2_2023), single run each:

test wall
tx_fuzzy, MAX_EXAMPLES=25 (default) 25.3 s
tx_fuzzy, MAX_EXAMPLES=1 (CI) 6.8 s
tx_edge 8.2 s
existing carstate_fuzzy, 300 examples (reference) 11.1 s

About 5 s of each number is fixed import and setup cost. At CI's MAX_EXAMPLES=1 the two tests together cost roughly
the same as the existing fuzz test; at the default 25 examples the fuzzer is about twice it. Route replay is the
price of catching the panda#1948 class, which a synthetic fuzzer cannot reach.

MAX_EXAMPLES is honoured as in the other fuzzy tests (CI runs with MAX_EXAMPLES=1).

Notes

  • Route replay with panda's stateful samples is what catches the class of bug in Toyota LTA: fix the way the torque sample_t's are used panda#1948 (the torque
    wind-down using a sample minimum); a synthetic fuzzer without real CAN history cannot reach it. The replay
    structure here follows the approach in test_models: fuzz the messages openpilot sends #3701, which I read before writing this; the engagement handling, the
    scenarios, the coverage accounting and the edge test are new.
  • I have no hardware. Everything here is host-side against libsafety, the same way the other panda safety tests
    in this file run.

@github-actions github-actions Bot added car related to opendbc/car/ car safety vehicle-specific safety code labels Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Car behavior report

Replays driving segments through this PR and compares the behavior to master.
Please review any changes carefully to ensure they are expected.

✅ No changes detected

Replays route CAN into both the car interface and panda safety so they
derive the same vehicle state, builds the CarControl controlsd would send
for fuzzed controls, and asserts every message the car controller emits
passes safety_tx_hook. Engagement is taken from panda's own controls_allowed
rather than forced, so a rejection is a limits mismatch and never an
artifact of the test making the two sides agree. Scenarios wind the rate
limiters to the actuator bound, toggle engagement mid-command, and skip
control frames so the real time rate checks see an irregular cadence.
Sent messages are counted per address so failures say what was exercised.

test_panda_safety_tx_edge covers the other direction: an accepted steering
command re-packed with the signal at the DBC field's extreme must be
rejected, with the unmodified re-pack asserted accepted first so a
rejection cannot be a round-trip artifact.

Also adds Fuzzy.real() to opendbc/testing.py.

Closes commaai/openpilot#32425

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

car related to opendbc/car/

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_models: add a test that fuzzes the tx messages

1 participant