Skip to content

[Fix] DDP init with device id - #2136

Merged
felixdittrich92 merged 1 commit into
mindee:mainfrom
felixdittrich92:minor-ddp-fixes
Sep 1, 2026
Merged

[Fix] DDP init with device id#2136
felixdittrich92 merged 1 commit into
mindee:mainfrom
felixdittrich92:minor-ddp-fixes

Conversation

@felixdittrich92

Copy link
Copy Markdown
Collaborator

This pull request improves distributed training reliability and device management for all detection, recognition, layout, and table training scripts. The main changes ensure that PyTorch's distributed process group is correctly initialized with device information and that CUDA-aware barriers are used for synchronization, which helps prevent deadlocks and device mismatch errors.

Distributed Training Initialization:

  • All train.py scripts (detection, recognition, layout, table) now initialize the process group with the device_id argument, ensuring each process is explicitly associated with the correct CUDA device. [1] [2] [3] [4]

Barrier Synchronization Improvements:

  • The barrier_download context manager in all ddp_utils.py modules (detection, recognition, layout, table) now uses a helper function to call dist.barrier with device_ids when using the NCCL backend and CUDA, ensuring correct device synchronization and avoiding potential deadlocks.

These changes collectively improve robustness and compatibility for multi-GPU distributed training across all relevant modules.

@felixdittrich92 felixdittrich92 self-assigned this Sep 1, 2026
@felixdittrich92 felixdittrich92 added this to the 1.2.0 milestone Sep 1, 2026
@felixdittrich92 felixdittrich92 added type: enhancement Improvement ext: references Related to references folder topic: text detection Related to the task of text detection topic: text recognition Related to the task of text recognition topic: table comprehension Related to table comprehension topic: layout detection Related to document layout detection labels Sep 1, 2026
@felixdittrich92
felixdittrich92 merged commit 77a599f into mindee:main Sep 1, 2026
54 checks passed
@felixdittrich92
felixdittrich92 deleted the minor-ddp-fixes branch September 1, 2026 10:14
@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.06%. Comparing base (d7a65fa) to head (9461c8c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2136      +/-   ##
==========================================
+ Coverage   97.01%   97.06%   +0.04%     
==========================================
  Files         169      169              
  Lines       10000    10000              
==========================================
+ Hits         9701     9706       +5     
+ Misses        299      294       -5     
Flag Coverage Δ
unittests 97.06% <ø> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ext: references Related to references folder topic: layout detection Related to document layout detection topic: table comprehension Related to table comprehension topic: text detection Related to the task of text detection topic: text recognition Related to the task of text recognition type: enhancement Improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant