Conversation
| import requests | ||
|
|
||
|
|
||
| def download_and_extract_datastack(datastack_url, extracted_datastack_dir): |
There was a problem hiding this comment.
This was not longer needed since we are downloading and extracting the datastack with curl and tar.
| dynamic = ["version"] # the version is provided dynamically by setuptools_scm | ||
| dependencies = [ | ||
| "flask", | ||
| "natcap.invest", |
There was a problem hiding this comment.
No longer a dependency since download_and_extract_datastack was removed
| micromamba activate invest_env | ||
| pip install setuptools setuptools_scm build cython babel | ||
| pip install --no-build-isolation git+https://github.com/natcap/pygeoprocessing.git | ||
| pip install --no-build-isolation git+https://github.com/emlys/invest.git@bugfix/2572 |
There was a problem hiding this comment.
Install the not-yet-released changes that fixed the memory leak issue
| project_id = var.project_id | ||
| region = var.region | ||
| startup_script = "startup_script.sh" | ||
| startup_script = "/home/bin/startup_script.sh" |
There was a problem hiding this comment.
Moved to an absolute path to prevent issues depending where this is run from.
| allow_automatic_updates = false | ||
| labels = var.labels | ||
| machine_type = "c2-standard-8" | ||
| machine_type = "c2-standard-4" |
There was a problem hiding this comment.
Downgraded for cost savings.
| } | ||
| } | ||
|
|
||
| CPUS_PER_TASK = 2 # number of cpus per task to submit to sbatch |
There was a problem hiding this comment.
c2-standard-4 has 4 vCPUs that correspond to only 2 physical cores. Slurm apparently counts the physical cores, and so we can't request more than 2.
There was a problem hiding this comment.
The relationship between the physical and virtual cores makes sense, and it also makes me wonder if we might be able to programmatically get the CPUs per task from the state of the environment somehow so that if we change the VM class, we don't also have to remember to update this module-level variable. Is that something we can easily get from the compute environment?
| json_path = f'{workspace_dir}/datastack/parameters.invest.json' | ||
| return textwrap.dedent(f"""\ | ||
| #!/bin/sh | ||
| #SBATCH --cpus-per-task={CPUS_PER_TASK} |
There was a problem hiding this comment.
By default --cpus-per-task is 1, hence why tasks were not being parallelized.
| region = var.region | ||
| reserved_ip_range = module.private_service_access.reserved_ip_range | ||
| zone = var.zone | ||
| size_gb = 2560 |
There was a problem hiding this comment.
Explicitly request the minimum filestore size, 2560 GB.
phargogh
left a comment
phargogh
left a comment
There was a problem hiding this comment.
Thanks @emlys ! I just had the one comment about possibly getting the number of CPUs per task from the environment, just to save us effort down the road. No worries if it isn't easy to extract that information, but it might be handy if we can!
| } | ||
| } | ||
|
|
||
| CPUS_PER_TASK = 2 # number of cpus per task to submit to sbatch |
There was a problem hiding this comment.
The relationship between the physical and virtual cores makes sense, and it also makes me wonder if we might be able to programmatically get the CPUs per task from the state of the environment somehow so that if we change the VM class, we don't also have to remember to update this module-level variable. Is that something we can easily get from the compute environment?
|
@phargogh good point! I looked into this and I think we can achieve the same thing by setting We may want to revisit this if we upgrade the compute nodes, or for a demo server we might want to limit each job to 1 CPU. Let's ignore the failing checks for now, I made some changes on another branch that messed up the authentication from Github Actions to the storage bucket. Should be fixed in the next PR. |
phargogh
left a comment
phargogh
left a comment
There was a problem hiding this comment.
Nice, simple solution! I like the --exclusive flag. Thanks for humoring my question!
2 participants
Fixes #33 by setting
--cpus-per-taskto 2.Throwing in some other small clean-up items, and downgrading some resources to save on costs.