-
Notifications
You must be signed in to change notification settings - Fork 530
[S#3543] Piece-wise linear compression of column groups first working prototype #2415 #2420
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 15 commits
Commits
Show all changes
38 commits
Select commit
Hold shift + click to select a range
163e4e4
wip:SYSTEMDS-3543
MaryAbdull f5df4ea
Meine lokalen Änderungen
MaryAbdull f6500d1
Merge upstream/main mit meinen lokalen Änderungen
MaryAbdull 8f5c844
wip: test
MaryAbdull 11415fa
Test Fix
janniklinde 5301f8f
wip: test
MaryAbdull a31116d
Merge pull request #1 from janniklinde/MaryamMain
MaryAbdull d63aae8
fix: Methods and testing
MaryAbdull 9600179
Merge remote-tracking branch 'origin/main'
MaryAbdull 78460b5
wip: decompressing
MaryAbdull f42b766
add: Enum Compressiontype piecewiselinear
MaryAbdull 47256c0
add: include functionality of piecewise linear compression
MaryAbdull 505c0cc
add: Comment
MaryAbdull 103abd8
add: dispatch test and remove unused imports
MaryAbdull 31b957d
fix: reformat code mit Eclipse XML Profile
MaryAbdull 0faa2f8
wip: fix formattaing
MaryAbdull 698a942
fix: reverted file
MaryAbdull 898af68
rm: comment reformatted and add targetloss handling
MaryAbdull d8ebc9f
fix: reverted file and add enum CompressionTypepiecewiseLinear
MaryAbdull 36d3186
fix: reverted file
MaryAbdull a0d08d7
fix: repeated compression on every column
MaryAbdull dfe2eee
add: utils, methods to calculate piecewiseLinearCompression
MaryAbdull 9e0d18b
wip: clear up tests
MaryAbdull abeced4
fix: revert pom.xml
MaryAbdull fc528ae
rm files
MaryAbdull 7b920c1
add: CompressionType and ColGroupType PiecewiseLinear
MaryAbdull bdb1550
add: second compression Method
MaryAbdull fd8eaf1
fix: rename second compression Method
MaryAbdull a19ef6a
add: second compression method
MaryAbdull 6006168
add: computing methods and operations
MaryAbdull 8b18910
wip: removed Test, used more on random generated matrices, add new ones
MaryAbdull 7f29223
fix: put operation tests in a seperate file
MaryAbdull 0b0a4fb
fix: bugfixes in methods and documentation
MaryAbdull d2014d2
fix: extract methods, refactor tests
MaryAbdull fc90898
fix: refactoring and documentation
MaryAbdull 739e237
fix: refactoring
MaryAbdull 4bd8e0b
fix: refactoring
MaryAbdull affa40a
performance test
MaryAbdull File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| #!/bin/bash | ||
| # Standalone-Launcher für SystemDS | ||
|
|
||
| SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd) | ||
| JAR_FILE="$SCRIPT_DIR/../target/systemds-3.4.0-SNAPSHOT.jar" | ||
|
|
||
| if [ ! -f "$JAR_FILE" ]; then | ||
| echo "ERROR: Standalone JAR nicht gefunden: $JAR_FILE" | ||
| exit 1 | ||
| fi | ||
|
|
||
| java -cp "$JAR_FILE" org.apache.sysds.api.DMLScript "$@" |
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please revert that file. You may keep your changes locally but these changes should not be pushed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -55,16 +55,16 @@ public class CompressionSettings { | |
| /** | ||
| * The sampling ratio used when choosing ColGroups. Note that, default behavior is to use exact estimator if the | ||
| * number of elements is below 1000. | ||
| * | ||
| * | ||
| * DEPRECATED | ||
| */ | ||
| public final double samplingRatio; | ||
|
|
||
| /** | ||
| * The sampling ratio power to use when choosing sample size. This is used in accordance to the function: | ||
| * | ||
| * | ||
| * sampleSize += nRows^samplePower; | ||
| * | ||
| * | ||
| * The value is bounded to be in the range of 0 to 1, 1 giving a sample size of everything, and 0 adding 1. | ||
| */ | ||
| public final double samplePower; | ||
|
|
@@ -114,8 +114,9 @@ public class CompressionSettings { | |
| /** | ||
| * Transpose input matrix, to optimize access when extracting bitmaps. This setting is changed inside the script | ||
| * based on the transposeInput setting. | ||
| * | ||
| * This is intentionally left as a mutable value, since the transposition of the input matrix is decided in phase 3. | ||
| * | ||
| * This is intentionally left as a mutable value, since the transposition of the input matrix is decided in phase | ||
| * 3. | ||
| */ | ||
| public boolean transposed = false; | ||
|
|
||
|
|
@@ -135,6 +136,21 @@ public class CompressionSettings { | |
|
|
||
| public final boolean preferDeltaEncoding; | ||
|
|
||
| /** | ||
| * Ziel-Gesantverlust für piecewise Lineace Komocession• Interpretation: maximal entaubter Alobaler MSE pro Went in | ||
| * der Sealte. O.O ~ quasi verlustfrei, viele Segmente >0 ~ mehr Approximation entaubt, weniger Segmente | ||
| */ | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Weird comment |
||
| private double piecewiseTargetLoss = Double.NaN; | ||
|
|
||
| public void setPiecewiseTargetLoss(double piecewiseTargetLoss) { | ||
| this.piecewiseTargetLoss = piecewiseTargetLoss; | ||
|
|
||
| } | ||
|
|
||
| public double getPiecewiseTargetLoss() { | ||
| return piecewiseTargetLoss; | ||
| } | ||
|
|
||
| protected CompressionSettings(double samplingRatio, double samplePower, boolean allowSharedDictionary, | ||
| String transposeInput, int seed, boolean lossy, EnumSet<CompressionType> validCompressions, | ||
| boolean sortValuesByLength, PartitionerType columnPartitioner, int maxColGroupCoCode, double coCodePercentage, | ||
|
|
@@ -161,7 +177,7 @@ protected CompressionSettings(double samplingRatio, double samplePower, boolean | |
| this.sdcSortType = sdcSortType; | ||
| this.scaleFactors = scaleFactors; | ||
| this.preferDeltaEncoding = preferDeltaEncoding; | ||
|
|
||
| if(!printedStatus && LOG.isDebugEnabled()) { | ||
| printedStatus = true; | ||
| LOG.debug(this.toString()); | ||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file should not be part of the PR. You can keep it locally but you should untrack it and not add it to your commits. You could use
git rm --cached bin/systemds-standalone.sh.