perf(embedding): use a gather-optimal device layout for nn.Embedding tables - #291
Open
ani300 wants to merge 1 commit into
Open
perf(embedding): use a gather-optimal device layout for nn.Embedding tables#291ani300 wants to merge 1 commit into
ani300 wants to merge 1 commit into
Conversation
…tables nn.Embedding weights are read as a gather (indexed by token id along the vocab/leading dim), not as a matmul. Previously they were merely excluded from the row-major matmul SpyreTensorLayout and left on the default layout. This instead gives each embedding table a gather-optimal "indirect access" layout: vocab dim outermost and the hidden dim split into BLOCK_SIZE-element sticks (device dims [rows, D // BLOCK_SIZE, BLOCK_SIZE]), built via the 3-arg device-dims SpyreTensorLayout overload with the device dtype from get_device_dtype(). When the hidden dim is not a multiple of BLOCK_SIZE the sticks can't tile it, so we warn and fall back to the default layout — the table still loads and runs, just without the gather optimization. Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>
Closed
1 task
Contributor
|
Thanks @ani300, how would this sync with #280, which basically plans to remove all hf-adapter-side layouting code? Can the embedding layouting sit in torch-spyre as well, or would we need to keep the hf-adapter-side layouting code for the embeddings specifically (cc @vinithakv) |
Contributor
Author
|
I can port this to the torch-spyre adapter instead, no problem. I'll move the PR there. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nn.Embeddingweights on Spyre are read as a gather (indexed by token id along the vocab/leading dim), not as a matmul. Today they are merely excluded from the row-major matmulSpyreTensorLayoutand left on the default layout — correct, but it leaves the gather unoptimized.This PR gives each embedding table a gather-optimal "indirect access" device layout instead:
BLOCK_SIZE-element sticks,i.e. device dims
[rows, D // BLOCK_SIZE, BLOCK_SIZE], built via the 3-arg device-dimsSpyreTensorLayoutoverload with the device dtype fromget_device_dtype()(not the hosttorch.dtype).What changed
hf_adapters/hf_common.py:_move_to_spyre_with_layoutnow chooses a device layout per weight: a gather-optimal layout for embedding tables, the row-major layout for 2-D matmul weights, and the default layout for everything else._embedding_layout(t)builds the indirect-access layout for a table._embedding_param_idsand the_alloc_on_spyrebranch/comments updated to reflect that embedding tables now get an active layout rather than being skipped.Fallback behavior
When the hidden dim
Dis not a multiple ofBLOCK_SIZE, the sticks can't tile the hidden dim. In that case wewarnings.warnand fall back to the default layout (None) — the table still loads and runs correctly, just without the gather optimization.Scope
Single self-contained change to the layout-selection logic. No API changes; models that previously loaded still load.
Testing
tests/spyre/test_e2e_token_compare_spyre.py) — verify layout applies and outputs match reference.