Allow setting the driver memory of the Docker image - #262
Merged
Conversation
The entrypoint hardcoded --driver-memory 4g, which could not be overridden from the command line: SPARK_SUBMIT_OPTS=-Xmx.. is emitted before the -Xmx derived from --driver-memory and so loses, and spark.driver.extraJavaOptions rejects -Xmx outright. The only way was to bypass the entrypoint entirely. The driver memory now comes from SPRUCE_DRIVER_MEMORY, defaulting to 4g as before, and is documented in the Docker tutorial. Also updates the S3 how-to, whose Docker example still passed a full spark-submit command as arguments; those have been arguments of the SPRUCE job itself since the entrypoint was introduced.
jnioche
added
docker
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Docker entrypoint hardcodes
--driver-memory 4g, and there is no way to raise it from thedocker runcommand line:SPARK_DRIVER_MEMORYloses to the explicit--driver-memoryflag;SPARK_SUBMIT_OPTS=-Xmx16gis emitted before the-Xmx4gderived from--driver-memory, so the JVM takes the 4g;spark.driver.extraJavaOptionsrejects-Xmxoutright.The only workaround was
--entrypoint /opt/spark/bin/spark-submitplus the full spark-submit command.The driver memory is now read from
SPRUCE_DRIVER_MEMORY, defaulting to4gso existing invocations are unchanged:docker run --rm -v $(pwd):/workspace -w /workspace \ -e SPRUCE_DRIVER_MEMORY=16g \ ghcr.io/digitalpebble/spruce \ -i curs -o outputVerified with
SPARK_PRINT_LAUNCH_COMMAND=1on an image built from this branch: no variable gives-Xmx4g,SPRUCE_DRIVER_MEMORY=9ggives-Xmx9g.Documentation: a new Allocating more memory section in the Docker tutorial, including the caveats that the container must actually be allowed that much memory and that
report/dashboardignore the variable.Also fixes the Docker example in the S3 how-to, which still passed a whole
spark-submitinvocation as arguments — since the entrypoint was introduced those are passed to the SPRUCE job instead, so the example could not work as written.