Add Vulkan GPU

[phreed]

This will form a separate GPU version.
Augments ROCm and CUDA versions.

[phreed]

Feasibility findings

Investigated against the current main (88738e9). Short version: possible, but a research project rather than a port — and the obstacle is not where you would expect.

The custom kernel is not the blocker

The library contains exactly one custom op, ms_deform_attn. PR IDEA-Research#449 (ROCm) also left a pure-PyTorch fallback built on F.grid_sample, reachable with GROUNDINGDINO_MSDA_FORCE_FALLBACK=1.

So a Vulkan port would not require writing a deformable-attention compute shader. The question reduces to: does PyTorch itself run on Vulkan?

It does not, in any usable form

On torch 2.13:

torch.is_vulkan_available()      -> False
torch.backends.vulkan            -> absent
torch.empty(1, device="vulkan")  -> NotImplementedError:
    Could not run 'aten::empty.memory_format' with arguments from the 'Vulkan' backend

The dispatch key still exists but no ops are registered in the standard wheel; it needs a source build with USE_VULKAN=1. Even then that backend is mobile-oriented, aimed at CNNs, and has been de-emphasised in favour of ExecuTorch.

What the model would demand of it

I instrumented a full forward pass (SwinT config, CPU) with a TorchDispatchMode recorder:

• 90 distinct aten ops
• 5,909 op invocations

The ones a Vulkan backend is least likely to handle:

Op	Calls	Why it is hard
aten._local_scalar_dense	228	this is .item() — 228 device→host syncs per forward. Even with complete op coverage this alone would likely make Vulkan slower than CPU
aten.grid_sampler_2d	48	the MsDeformAttn fallback itself
aten.native_layer_norm	123
aten.masked_fill(_)	55
aten.roll	12	Swin shifted windows
aten.index.Tensor	12
aten.constant_pad_nd	12
aten.cumsum	9
aten.upsample_nearest2d	4
aten.gather / aten.embedding	3 each
aten.topk	1
aten.nonzero	1	data-dependent output shape — essentially incompatible with a static-shape mobile backend

The single nonzero (in generate_masks_with_special_tokens_and_transfer_map) and the 228 .item() calls are the real obstacles. They also make torch.export awkward, which is the entry point to ExecuTorch.

Realistic routes, if Vulkan specifically is the goal

1. IREE (MLIR → SPIR-V). Most likely to work for a model this complex. ONNX → StableHLO → Vulkan; export_onnx/ already provides the first hop.
2. ExecuTorch Vulkan delegate. The actively developed path; unsupported ops fall back to CPU. Data-dependent shapes are the risk at export time.
3. ncnn. Mature Vulkan backend, consumes ONNX. GridSample support is the first thing to check.
4. Native torch Vulkan backend. Not viable today.

A cheaper option worth considering first

If the underlying goal is "run on non-NVIDIA hardware" rather than Vulkan as such, note that torch.xpu already exists in torch 2.13 (Intel oneAPI/SYCL), and this codebase is only lightly CUDA-hardcoded — 3 files, and misc.py's occurrences are all distributed-training helpers irrelevant to inference. The substantive ones are:

• load_model(..., device="cuda") and the other device: str = "cuda" defaults in util/inference.py
• torch.cuda.is_available() in the use_custom_kernel guard in models/GroundingDINO/ms_deform_attn.py

Making those device-agnostic would let XPU and MPS work through the existing fallback, and is a prerequisite for any Vulkan route anyway. That is a far smaller piece of work, and it is a sensible first step for this issue.

Caveat

This is analysis only. Nothing Vulkan-related has been implemented or run, and no Vulkan-capable hardware was available.