Skip to content

test_models: fuzz the tx path through panda safety - #3730

Open
597226617 wants to merge 1 commit into
commaai:masterfrom
597226617:test/fuzz-tx-safety
Open

test_models: fuzz the tx path through panda safety#3730
597226617 wants to merge 1 commit into
commaai:masterfrom
597226617:test/fuzz-tx-safety

Conversation

@597226617

Copy link
Copy Markdown

Companion to openpilot #32425.

#30443 set up the rx-side comparison — openpilot and panda parsing the same CAN messages with the same validity checks. This does the same for tx: every message the openpilot controller sends has to pass panda's safety_tx_hook, and panda must not wave through DBC extreme values. The goal is to catch things like panda#1948, where the two limiters drift apart and one rejects what the other just sent.

Three parts:

  • test_panda_safety_tx_fuzzy — drives CarInterface and the safety rx/tx hooks with the same fuzzed input state (using the existing deterministic Fuzzy helper, FUZZ_SEED/FUZZ_EXAMPLE reproducible, MAX_EXAMPLES=1 in CI like the rest of the suite)
  • DBC extreme-value tests — every signal at its DBC min/max, including the asymmetric ones
  • Fuzzy.float extension — interval exclusion so endpoints aren't starved

Two things fell out while testing:

  1. VW MEB sends a curvature jump when falling out of ACC (steer_state mismatch — the controller switches mode before the curvature clamp adapts, panda's jerk window rejects it). Fixed in carcontroller here; same class of bug as what's in flight for test_models: fuzz the messages openpilot sends #3701.
  2. Honda has no steering limit checks at all in engaged state — raw steer values (32767) pass straight through where every other torque brand gets clamped. Left as an explicit known-gap skip with a pointer comment; the actual limits are a platform decision, not something I should invent.

Tests run locally end to end through libsafety via cffi. Determinism checked (same seed, same hash, twice).

Mirrors the rx-side comparison from #30443 (openpilot PR): every
message the openpilot controller sends must pass panda's safety
tx hook, and panda must not let DBC extreme values through. Also
fixes a curvature jump on VW MEB when falling out of ACC.
@github-actions github-actions Bot added car related to opendbc/car/ volkswagen labels Sep 9, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for contributing to opendbc! In order for us to review your PR as quickly as possible, check the following:

  • Convert your PR to a draft unless it's ready to review
  • Read the contributing docs
  • Before marking as "ready for review", ensure:
    • the goal is clearly stated in the description
    • all the tests are passing
    • include a route or your device' dongle ID if relevant

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Car behavior report

Replays driving segments through this PR and compares the behavior to master.
Please review any changes carefully to ensure they are expected.

Testing 130 segments for: VOLKSWAGEN_ARTEON_MK1, VOLKSWAGEN_ATLAS_MK1, VOLKSWAGEN_CRAFTER_MK2, VOLKSWAGEN_GOLF_MK7, VOLKSWAGEN_JETTA_MK7, VOLKSWAGEN_PASSAT_MK8, VOLKSWAGEN_PASSAT_NMS, VOLKSWAGEN_POLO_MK6, VOLKSWAGEN_TAOS_MK1, VOLKSWAGEN_TIGUAN_MK2, VOLKSWAGEN_TOURAN_MK2, VOLKSWAGEN_TRANSPORTER_T61, VOLKSWAGEN_TROC_MK1

✅ 0 changed, 128 passed, 2 errors

Show errors

Traceback (most recent call last):
  File "/home/runner/work/opendbc/opendbc/opendbc/car/tests/car_diff.py", line 86, in process_segment
    can_msgs = load_can_messages(seg)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/opendbc/opendbc/opendbc/car/tests/car_diff.py", line 56, in load_can_messages
    url = get_url(f"{parts[0]}/{parts[1]}", parts[2])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/opendbc/opendbc/.venv/lib/python3.12/site-packages/comma_car_segments/__init__.py", line 91, in get_url
    return get_repo_url(f"segments/{route.replace('|', '/')}/{segment}/{file}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/opendbc/opendbc/.venv/lib/python3.12/site-packages/comma_car_segments/__init__.py", line 80, in get_repo_url
    assert response.status_code == 200
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

car related to opendbc/car/ volkswagen

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant