TP enablement - #234
Open
arielge wants to merge 3 commits into
Open
Conversation
|
I have a PR open to add asyncOp support to the torch-spyre backend: With that PR, I got past the asyncOp assertion and hit a new one that doesn't look Spyre Comms-related. #----------------------------------------------------------------------
# Compiling the Model
#----------------------------------------------------------------------
Compile time: 0.0s
#----------------------------------------------------------------------
# Running Model: Warmup
# Prompt: Provide a list of instructions for preparing chicken soup.
#----------------------------------------------------------------------
/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/hf-adapters/hf_adapters/hf_granite.py:119: FallbackWarning: aten.embedding.default is falling back to cpu
h = backbone.embed_tokens(input_ids)
/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/hf-adapters/hf_adapters/hf_granite.py:119: FallbackWarning: aten.embedding.default is falling back to cpu
h = backbone.embed_tokens(input_ids)
/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/hf-adapters/hf_adapters/hf_granite.py:119: FallbackWarning: aten.embedding.default is falling back to cpu
h = backbone.embed_tokens(input_ids)
[rank0]: Traceback (most recent call last):
[rank0]: Traceback (most recent call last):
[rank0]: File "/root/project_src/test_granite.py", line 74, in <module>
[rank0]: outputs = compiled_model.generate(tokenizer,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/hf-adapters/hf_adapters/auto_spyre_model.py", line 248, in model_generate
[rank0]: return generate(module._run_forward, self, tokenizer, prompts, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/hf-adapters/hf_adapters/hf_common.py", line 1643, in generate
[rank0]: logits = run_forward_fn(
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/hf-adapters/hf_adapters/hf_granite.py", line 152, in _run_forward
[rank0]: h = _run_backbone_forward(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/hf-adapters/hf_adapters/hf_granite.py", line 125, in _run_backbone_forward
[rank0]: h, key_caches[i], value_caches[i] = compiled_block(
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1038, in compile_wrapper
[rank0]: raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1053, in _compile_fx_inner
[rank0]: raise InductorError(e, currentframe()).with_traceback(
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1037, in _compile_fx_inner
[rank0]: mb_compiled_graph = fx_codegen_and_compile(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1798, in fx_codegen_and_compile
[rank0]: return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1570, in codegen_and_compile
[rank0]: compiled_module = graph.compile_to_module()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_inductor/graph.py", line 2499, in compile_to_module
[rank0]: return self._compile_to_module()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_inductor/graph.py", line 2505, in _compile_to_module
[rank0]: self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
[rank0]: ^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib64/python3.12/site-packages/torch/_inductor/graph.py", line 2437, in codegen
[rank0]: self._update_scheduler()
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/torch-spyre/torch_spyre/_inductor/patches.py", line 120, in _spyre_update_scheduler
[rank0]: _pre_scheduling_pass(self)
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/torch-spyre/torch_spyre/_inductor/passes.py", line 386, in __call__
[rank0]: pass_fn(graph)
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/torch-spyre/torch_spyre/_inductor/propagate_layouts.py", line 1416, in propagate_spyre_tensor_layouts
[rank0]: op.layouts = compute_layouts(op, output, output_dep, args)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/torch-spyre/torch_spyre/_inductor/propagate_layouts.py", line 1091, in compute_layouts
[rank0]: result = _single_arg_op_layout(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/torch-spyre/torch_spyre/_inductor/propagate_layouts.py", line 472, in _single_arg_op_layout
[rank0]: in_device_coords = device_coordinates(stl, dep, None)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/torch-spyre/torch_spyre/_inductor/pass_utils.py", line 734, in device_coordinates
[rank0]: _check_stick_expr_supported(coords[-1], stl.elems_per_stick())
[rank0]: File "/root/multi-aiu-dev/_dev/sentient-ci-cd/_dev/sen_latest/torch-spyre/torch_spyre/_inductor/pass_utils.py", line 699, in _check_stick_expr_supported
[rank0]: raise Unsupported(
[rank0]: torch._inductor.exc.InductorError: Unsupported: Spyre backend does not support: Unexpected stick expression 1: expected Mod(var, 64), a bare variable, 0, or any of those with a constant offset |
This was referenced Jul 23, 2026
|
With the updated |
Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
Signed-off-by: Ariel Gera <ariel.gera1@ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starting to address #158, for now we have:
5.14.1, which raises on certain vocab sizes (🚨 Pass tp_plan from lm_heads huggingface/transformers#47253)from_pretrained()with TP plank_projshape and not the config, supporting the TP sharding scenario