Draft: Region per file storage strategy#287
Open
TheAssembler1 wants to merge 3 commits intohpc-io:developfrom
Open
Draft: Region per file storage strategy#287TheAssembler1 wants to merge 3 commits intohpc-io:developfrom
TheAssembler1 wants to merge 3 commits intohpc-io:developfrom
Conversation
b2a04b2 to
36cdffb
Compare
f98bb9e to
79a45b9
Compare
a3ae926 to
844ae19
Compare
Collaborator
Author
|
We might want to compare the performance between the storage strategies before merging. |
Member
|
@TheAssembler1 do you have any updates on this PR? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the following
enumso the storage strategy can be selected:The
STORE_REGION_BY_REGION_SINGLE_FILEis the default strategy. TheSTORE_FLATTENED_REGION_PER_FILEis the new strategy which stores each region of an object in a separate file. The region size the object is sliced into is decided in:By default it will try to slice the object into regions that are 4 MB in size by halving the largest dimension of the object iteratively until within the <= 4 MB.
This is set here
uint64_t max_bytes_per_file = 4ULL * 1024 * 1024;within thePDC_shrink_file_dimsfunction.