Request for DriveLM Dataset Data Extraction and Safety Inference Code

[yuep0824]

Hello Repo Maintainers,

First and foremost, I would like to express my sincere gratitude for your impactful work on autonomous driving safety. This repository has provided invaluable support for our research team’s exploration in the field of autonomous driving safety, and we have been actively building our research framework based on it.

During the process of reproducing the experiments and extending the research, we encountered a key challenge: we are currently lacking the source code related to DriveLM dataset data extraction and safety inference modules. These components are critical for us to fully validate the effectiveness of your proposed methods and conduct in-depth comparative experiments for our own research.

We would greatly appreciate it if you could share the corresponding source code (if available) or provide alternative solutions (such as recommended open-source tools, data processing pipelines, or technical references that can achieve similar functions). This will not only help us smoothly advance our research but also allow us to better understand and promote the application of your work.

Thank you again for your contributions to the community and your kind support. We look forward to your reply!

Best regards

[javyduck]

hi sorry for the late reply, for DriveLM dataset data extraction and safety inference modules
for the first one, we have included the raw dataset here: https://github.com/AI-secure/SafeAuto/tree/main/data/conversation/drivelm
for safety inference modules, we have incoprated it in our code, if you still have any problem on that, feel free to let me know!

[javyduck]

Hi, thanks for the interest in our work, and sorry for the late follow-up! To summarize the current status:

1. DriveLM data: the processed conversation data is included under data/conversation/drivelm, and the raw datasets can be downloaded with bash scripts/download_datasets.sh.
2. Safety inference: the PGM/MLN safety verification module is now fully released — see pgm/ (rules in pgm/config.py, training via python -m pgm.pgm --dataset bddx) and its integration into inference in llava/serve/eval_custom_predsig_bddx.py. The pretrained BDD-X PGM checkpoint is also included at pgm/ckpts/pgm/bddx_weights.npy on the latest main.
3. Predicate extraction: the BDD-X extraction pipeline (YOLO-based environment predicates + GPT-based action/control-signal mapping) is released as pgm/bddx_predicate_extractor.py. The DriveLM-specific extractor is not cleaned up yet (marked as TODO in the README) — we will try to release it as well.

Hope this helps your research, and feel free to follow up if anything is unclear!

[javyduck]

Hi again — good news: the DriveLM data extraction and safety inference modules are now fully released on the latest main! Concretely:

1. DriveLM predicate extraction: pgm/drivelm_predicate_extractor.py (YOLO traffic signs/lights + NuScenes map lane markings + behavior-option parsing). The extraction results are also released at data/extraction/drivelm/extraction_drivelm_{train,eval}.json, so you don't need the raw NuScenes data unless you want to regenerate them from scratch.
2. DriveLM PGM (MLN) safety inference: the original checkpoint is at pgm/ckpts/pgm/drivelm_weights.npy, training vectors at pgm/predicates/drivelm/train_pgm_vectors.pkl (retrain with python -m pgm.pgm --dataset drivelm --learning_rate 0.01), and the post-safety verification of the behavior answer is integrated into llava/serve/eval_custom_predsig_drivelm.py (run via bash scripts/eval_drivelm.sh).
3. Evaluation: python evaluate_drivelm.py --path <result_dir> now reports behavior/speed/steering accuracy against the released ground truth (data/drivelm_val_answers.json).

As a sanity check, applying the released PGM verification to the paper's DriveLM predictions improves behavior accuracy from 74.01% to 74.45%. Hope this unblocks your research — feel free to reach out if anything is unclear!