Introduce XPU scope profiler extending existing XPU profiler plugin by moksiuc · Pull Request #1174 · pytorch/kineto

moksiuc · 2025-11-13T09:04:28Z

Summary:

As XPU became a PyTorch built-in device, the profiler support is indispensable part of functionality completeness. In this PR, the XPU scope profiler is introduced by extending existing XPU profiler plugin. The XPU scope profiler is built on the foundation of intel PTI toolkit (https://github.com/intel/pti-gpu), and underlying SYCL runtime. It allows to gather XPU hardware metrics. The LIBKINETO_NOXPUPTI option is used to enable or disable the whole XPU profiler plugin during kineto build stage.

Changes:

Added new ActivityType : XPU_SCOPE_PROFILER, enabling the new scope profiler
Outputs XPU hardware metrics from the new scope profiler in Perfetto counters display mode ("C")
Added gtest

moksiuc · 2025-11-13T09:28:50Z

@EikanWang, @gujinghui

gujinghui · 2025-11-14T06:34:11Z

@moksiuc It's great that we are going to update our PTI integration code, and introduce new profiler path.
Could you help address below questions?

The alternative of ScopeProfiler is the RangeProfiler for CUDA? Looks like the RangeProfler is not enabled in PyTorch by default so far. Do you know why?
This PR is too huge to review. Can we split it to several PRs? For example, one PR for code refactor or cleanup per kineto or PTI changes, one or two PRs for ScopeProfiler, one PR for ChromeTraceLogger enhancement, and add test cases for each PRs.
BTW, CUDA provides CUDA_DRIVER activity to trace the driver actions. We should provide L0 actions as the counterpart, right? I remember, PTI should be able to do that. Do we have plan to cover it?

kineto/libkineto/src/ActivityType.cpp

Line 29 in 1e30d37

{"cuda_driver", ActivityType::CUDA_DRIVER},

moksiuc · 2025-11-14T13:53:03Z

The alternative of ScopeProfiler is the RangeProfiler for CUDA? Looks like the RangeProfler is not enabled in PyTorch by default so far. Do you know why?
It is enabled by providing experimental_config=_ExperimentalConfig(...). I don't know why it is this way but we are enabling our profiler the same way. One of the reasons may be that Range/Scope profiler requires parameters like HW metrics names that are passed through _ExperimentalConfig.

moksiuc · 2025-11-14T13:53:59Z

BTW, CUDA provides CUDA_DRIVER activity to trace the driver actions. We should provide L0 actions as the counterpart, right? I remember, PTI should be able to do that. Do we have plan to cover it?

kineto/libkineto/src/ActivityType.cpp

Line 29 in 1e30d37

{"cuda_driver", ActivityType::CUDA_DRIVER},

For sure not in this PR. I'll add this to our list of tasks.

moksiuc · 2025-11-17T10:35:03Z

This PR is too huge to review. Can we split it to several PRs? For example, one PR for code refactor or cleanup per kineto or PTI changes, one or two PRs for ScopeProfiler, one PR for ChromeTraceLogger enhancement, and add test cases for each PRs.

Extracted clean up and adding config for scope profiler to separate PR's.
This one should be much smaller afterwards.
Currently I don't see further areas of extracting separate PR's as what would remain is full scope profiler implementation with tests and we'd like not to introduce half of the implementation that is not working functionally.

- removed rangeEnabled - fix test to align to this removal - erase used kernelActivity from map

- place of config initialization - removal of passing unused C compiler flag into test cmake file

gujinghui

This PR is split to #1177, #1180, and more.

gujinghui · 2025-12-02T01:51:47Z

@moksiuc Let's close this PR.

moksiuc · 2025-12-03T07:48:58Z

@gujinghui this is the core of the scope profiler. When 2 smaller parts are merged this one would have only core profiler left.

This reverts commit 8d9b899.

This reverts commit d0a82c1.

This reverts commit 5230042.

gujinghui · 2026-04-10T06:22:52Z

@divyanshk @scotts all comments have been addressed. Could you help review again? Thanks.

This reverts commit f329732.

This reverts commit 3b2cb5a.

divyanshk

Thanks for addressing the comments! Looks good to me!

moksiuc · 2026-04-17T09:08:53Z

Failure in the test
Linux CUDA build and test / pr-test / linux-cuda-build-and-test (pull_request)
seems unrelated to my change:
=================================== FAILURES ===================================
_______________________ TestProfiler.test_dynamic_toggle _______________________
Traceback (most recent call last):
File "/opt/conda/lib/python3.11/unittest/case.py", line 57, in testPartExecutor
yield
File "/opt/conda/lib/python3.11/unittest/case.py", line 623, in run
self._callTestMethod(testMethod)
File "/opt/conda/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
if method() is not None:
^^^^^^^^
File "/pytorch/kineto/pytorch/torch/testing/_internal/common_utils.py", line 3448, in wrapper
method(*args, **kwargs)
File "/pytorch/kineto/pytorch/torch/testing/_internal/common_utils.py", line 1821, in wrapper
fn(*args, **kwargs)
File "/pytorch/kineto/pytorch/test/profiler/test_profiler.py", line 2312, in test_dynamic_toggle
self.assertTrue(all(device not in e.name for e in p1.events()))
File "/opt/conda/lib/python3.11/unittest/case.py", line 715, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true

gujinghui · 2026-04-20T05:25:14Z

@scotts Could you help review this PR again? We already addressed all comments. Thanks.

aostrowski-hbn · 2026-04-29T07:51:23Z

@scotts @divyanshk Can we merge?

meta-codesync · 2026-04-29T17:44:45Z

@divyanshk has imported this pull request. If you are a Meta employee, you can view this in D103032091.

meta-codesync · 2026-04-30T04:59:17Z

@divyanshk merged this pull request in d636a94.

Includes the following commits: - Replace LIBKINETO_NO* build flags with KINETO_BACKEND (pytorch/kineto#1388) 7739225 - Add git fetch <path_to_kineto_repo> main (pytorch/kineto#1384) 63cc35e - Daily `arc lint --take CLANGFORMAT` (pytorch/kineto#1386) b40959c - Introduce XPU scope profiler extending existing XPU profiler plugin (pytorch/kineto#1174) d636a94 - Refactor CI to be more fine-grain (pytorch/kineto#1385) f78048c Pull Request resolved: #182973 Approved by: https://github.com/divyanshk

Includes the following commits: - Replace LIBKINETO_NO* build flags with KINETO_BACKEND (pytorch/kineto#1388) 7739225 - Add git fetch <path_to_kineto_repo> main (pytorch/kineto#1384) 63cc35e - Daily `arc lint --take CLANGFORMAT` (pytorch/kineto#1386) b40959c - Introduce XPU scope profiler extending existing XPU profiler plugin (pytorch/kineto#1174) d636a94 - Refactor CI to be more fine-grain (pytorch/kineto#1385) f78048c Pull Request resolved: pytorch#182973 Approved by: https://github.com/divyanshk

scope profiler squashed

52c714a

meta-cla Bot added the cla signed label Nov 13, 2025

moksiuc changed the title ~~scope profiler squashed~~ Introduce XPU scope profiler extending existing XPU profiler plugin Nov 13, 2025

Type fix

0cbe479

gujinghui reviewed Nov 14, 2025

View reviewed changes

Comment thread libkineto/src/plugin/xpupti/FindSYCLToolkit.cmake Outdated

This was referenced Nov 14, 2025

Code cleanup of XPU profiler for incoming scope profiler #1177

Closed

Add config for incoming XPU scope profiler #1180

Closed

moksiuc added 2 commits November 17, 2025 13:20

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

7c98f84

Fixes

8438227

- removed rangeEnabled - fix test to align to this removal - erase used kernelActivity from map

gujinghui approved these changes Nov 18, 2025

View reviewed changes

moksiuc added 3 commits November 18, 2025 15:29

Code review fixes

fd2a6e1

- place of config initialization - removal of passing unused C compiler flag into test cmake file

Fix clang build on PT level

88082d5

Fix clang build on PT level 2nd approach

04f305c

gujinghui suggested changes Nov 19, 2025

View reviewed changes

moksiuc added 5 commits November 19, 2025 12:41

lintrunner

dbc709f

Improve json for presentation in perfetto

ad4b1dd

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

e050a30

Fix gtest after json update

37eafea

Put correct oneapi version in error msg

a266b04

moksiuc marked this pull request as ready for review November 24, 2025 10:01

Fix correct PTI and OneApi version

4a70122

sraikund16 and others added 2 commits December 17, 2025 13:19

lint

535b19e

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

b639a8c

moksiuc added 2 commits March 30, 2026 17:04

Rename V2 -> Scope according to review comments

a312ef6

Reapply "unformat"

d0a82c1

This reverts commit 8d9b899.

gujinghui mentioned this pull request Apr 9, 2026

Add support for oneCCL and Synchronization events. #1335

Open

moksiuc added 4 commits April 9, 2026 15:43

Revert "Reapply "unformat""

5230042

This reverts commit d0a82c1.

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

dfbd801

Align to new kineto code in output_json.cpp

ef7102f

Reapply "unformat"

f329732

This reverts commit 5230042.

gujinghui approved these changes Apr 10, 2026

View reviewed changes

moksiuc requested review from divyanshk and scotts April 10, 2026 08:03

moksiuc added 3 commits April 16, 2026 11:17

Revert "unformat"

3b2cb5a

This reverts commit f329732.

Merge branch 'refs/heads/main' into moksiuci_6674_scope_profiler

c27b580

Reapply "unformat"

840675e

This reverts commit 3b2cb5a.

scotts added the ciflow/xpu label Apr 16, 2026

divyanshk approved these changes Apr 16, 2026

View reviewed changes

meta-codesync Bot closed this in d636a94 Apr 30, 2026

facebook-github-tools Bot added the Merged label Apr 30, 2026

scotts mentioned this pull request May 8, 2026

Update third_party/kineto submodule to 7739225 pytorch/pytorch#182973

Closed

aostrowski-hbn mentioned this pull request May 13, 2026

Enable kineto XPU scope profiler extending existing XPU profiler plugin pytorch/pytorch#165766

Open

moksiuc deleted the moksiuci_6674_scope_profiler branch May 18, 2026 17:09

Conversation

moksiuc commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moksiuc commented Nov 13, 2025

Uh oh!

Uh oh!

gujinghui commented Nov 14, 2025

Uh oh!

moksiuc commented Nov 14, 2025

Uh oh!

moksiuc commented Nov 14, 2025

Uh oh!

moksiuc commented Nov 17, 2025

Uh oh!

gujinghui left a comment

Choose a reason for hiding this comment

Uh oh!

gujinghui commented Dec 2, 2025

Uh oh!

moksiuc commented Dec 3, 2025

Uh oh!

gujinghui commented Apr 10, 2026

Uh oh!

divyanshk left a comment

Choose a reason for hiding this comment

Uh oh!

moksiuc commented Apr 17, 2026

Uh oh!

gujinghui commented Apr 20, 2026

Uh oh!

aostrowski-hbn commented Apr 29, 2026

Uh oh!

meta-codesync Bot commented Apr 29, 2026

Uh oh!

meta-codesync Bot commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

moksiuc commented Nov 13, 2025 •

edited

Loading