feat(kernel range reader): Add buffer pooling support for large reads on regional buckets#4848
Draft
abhishek10004 wants to merge 5 commits into
Draft
feat(kernel range reader): Add buffer pooling support for large reads on regional buckets#4848abhishek10004 wants to merge 5 commits into
abhishek10004 wants to merge 5 commits into
Conversation
…eader (#4845) Adding support for vectored read in Kernel Range reader implementation.
…abhishek/sync_pool
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Description
This PR introduces buffer pooling for large and vectored read operations to optimize memory usage and reduce garbage collection overhead. By reusing allocated buffers via a sync pool, we minimize memory churn during sequential and large file reads.
Note
Buffer pooling support for larger reads is only supported for regional buckets when kernel reader is enabled.
Key Changes
Buffer Pooling & Lifecycle Management (
internal/fs/fs.go)readBufferPoolinto read operations to dynamically allocate and recycle byte buffers for read requests (supported only for regional buckets with kernel reader enabled).Buffer Limiting Utility (
internal/util/util.go)LimitBuffers(buffers [][]byte, limit int) [][]byte.Testing (
internal/fs/large_read_regional_test.go,internal/util/util_test.go)LimitBufferscovering boundary conditions, multiple buffer spans, and edge cases.large_read_regional_test.go) verifying successful vectored reads on regional buckets and confirming proper fallback/error behavior (ENOTSUP) for zonal buckets or when kernel reader support is disabled.Link to the issue in case of a bug fix.
b/530765312
Testing details
Any backward incompatible change? If so, please explain.