Implemented array-job-based SLURM integration, deprecated old version

contrib/slurm-benchmark.py

@@ -14,6 +14,8 @@
 import os
 import sys
 
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
+
 import benchexec.benchexec
 import benchexec.tools
 import benchexec.util
@@ -43,6 +45,12 @@ def create_argument_parser(self):
             action="store_true",
             help="Use SLURM to execute benchmarks.",
         )
+        slurm_args.add_argument(
+            "--slurm-array",
+            dest="slurm_array",
+            action="store_true",
+            help="Use SLURM array jobs to execute benchmarks.",
+        )
         slurm_args.add_argument(
             "--singularity",
             dest="singularity",
@@ -61,13 +69,65 @@ def create_argument_parser(self):
             dest="retry",
             type=int,
             default="0",
-            help="Retry killed jobs this many times. Use -1 for unbounded retry attempts.",
+            help="Retry killed jobs this many times. Use -1 for unbounded retry attempts (cannot be used with --slurm-array).",
+        )
+
+        slurm_args.add_argument(
+            "--aggregation-factor",
+            dest="aggregation_factor",
+            type=int,
+            default="10",
+            help="Aggregation factor for batch jobs (this many tasks will run in a single SLURM job).",
+        )
+        slurm_args.add_argument(
+            "--batch-size",
+            dest="batch_size",
+            type=int,
+            default="5000",
+            help="Split run sets into batches of at most this size. Helpful in avoiding errors with script sizes.",
+        )
+        slurm_args.add_argument(
+            "--parallelization",
+            dest="concurrency_factor",
+            type=int,
+            default="4",
+            help="Run this many tasks at once in one job.",
+        )
+        slurm_args.add_argument(
+            "--overtime-factor",
+            dest="overtime_factor",
+            type=float,
+            default="1.1",
+            help="Factor which by to scale timelimits to overapproximate CPU time limit with walltime limit.",
+        )
+        slurm_args.add_argument(
+            "--continue-interrupted",
+            dest="continue_interrupted",
+            action="store_true",
+            help="Continue a previously interrupted job.",
+        )
+        slurm_args.add_argument(
+            "--copy-tool",
+            dest="copy_tool",
+            action="store_true",
+            help="Make a copy of the tool folder in the container.",
+        )
+        slurm_args.add_argument(
+            "--generate-only",
+            dest="generate_only",
+            action="store_true",
+            help="Only generate the SLURM array description, don't run it.",
         )
 
         return parser
 
     def load_executor(self):
-        if self.config.slurm:
+        if self.config.slurm_array:
+            from slurm import arrayexecutor as executor
+        elif self.config.slurm:
+            logging.error(
+                "Single-job-based SLURM-integration is no longer supported. Use --slurm-array instead."
+            )
             from slurm import slurmexecutor as executor
         else:
             logging.warning(

contrib/slurm/README-old.md

@@ -0,0 +1,94 @@
+<!--
+This file is part of BenchExec, a framework for reliable benchmarking:
+https://github.com/sosy-lab/benchexec
+
+SPDX-FileCopyrightText: 2021 Dirk Beyer <https://www.sosy-lab.org>
+SPDX-FileCopyrightText: 2024 Levente Bajczi
+SPDX-FileCopyrightText: Critical Systems Research Group
+SPDX-FileCopyrightText: Budapest University of Technology and Economics <https://www.ftsrg.mit.bme.hu>
+
+SPDX-License-Identifier: Apache-2.0
+-->
+# BenchExec Extension for Benchmarking via SLURM
+
+> [!CAUTION]
+> This, single-job-based SLURM integration is no longer maintained. For the maintained, array-based version's documentation, see [README.md](./README.md)
+
+This Python script extends BenchExec, a benchmarking framework, to facilitate benchmarking via SLURM, optionally using a Singularity container.
+
+In case of problems, please tag in an [issue](https://github.com/sosy-lab/benchexec/issues/new/choose): [Levente Bajczi](https://github.com/leventeBajczi) (@leventeBajczi).
+
+## Preliminaries
+
+* [SLURM](https://slurm.schedmd.com/documentation.html) is an open-source job scheduling and workload management system used primarily in high-performance computing (HPC) environments.
+* [Singularity](https://docs.sylabs.io/guides/latest/user-guide/) is a containerization platform designed for scientific and high-performance computing (HPC) workloads, providing users with a reproducible and portable environment for running applications and workflows.
+
+## Requirements
+
+* SLURM, tested with `slurm 22.05.7`, should work within `22.x.x`
+* Singularity (optional), tested with `singularity-ce version 4.0.1`, should work within `4.x.x`
+
+## Usage
+1. Run the script with Python 3:
+   ```
+   python3 $BENCHEXEC_FOLDER/contrib/slurm-benchmark.py [options]
+   ```
+   Options:
+   - `--slurm`: Use SLURM to execute benchmarks. Will revert to regular (local) benchexec if not given. 
+   - `--singularity <path_to_sif>`: Specify the path to the Singularity .sif file to use. See usage later.
+   - `--scratchdir <path>`: Specify the directory for temporary files. The script will use this parameter to create temporary directories for file storage per-run, which get discarded later. By default, this is the CWD, which might result in temporary files being generated by the thousands in the working directory. On some systems, this must be on the same mount, or even under the same hierarchy as the current directory. Must exist, be writable, and be a directory.
+   - `--retry-killed <N>`: Retry killed jobs (e.g., due to SLURM errors) this many times. Use -1 for unbounded retry attempts.
+   - `-N <N>`: Specify the factor of parallelism, i.e., how many instances to start at a time. Tested with up to `1000`, probably works with much higher values as well.
+
+## Overview of the Workflow
+
+This works similarly to BenchExec, however, instead of delegating each run to `runexec`, it delegates to `srun` from SLURM.
+
+1. If the `--singularity` option is given, the script wraps the command to run in a container. This is useful for dependency management (in most HPC environments, arbitrary package installations are frowned upon). For a simple container, use the following: 
+
+    ```singularity
+    BootStrap: docker
+    From: ubuntu:22.04
+
+    %post
+    apt -y update
+    apt -y install openjdk-17-jre-headless libgomp1 libmpfr-dev fuse-overlayfs
+    ```
+
+    Use `singularity build [--remote / --fakeroot] --fix-perms <name>.sif <name>.def` to build the container.
+
+    Notice the `fuse-overlayfs` package. That is mandatory for the overlay filesystem to work properly.
+
+    The script parameterizes `singularity exec` with the following params:
+    * `-B $PWD:/lower`: Bind the working directory to `/lower` (could be read-only)
+    * `--no-home`: Do not bind the home directory
+    * `-B {tempdir}:/overlay`: Bind the temporary directory to `/overlay` (must be writeable)
+    * `--fusemount  "container:fuse-overlayfs -o lowerdir=/lower -o upperdir=/overlay/upper -o workdir=/overlay/work $HOME"`: mount an overlay filesystem at $HOME, where modifications go in the temp dir but files can be read from the current dir
+
+    We also wrap this command inside the container using `bash -c "{command} && echo 0 > exitcode || echo $? > exitcode` to save the exitcode of the process, _and_ always have 0 as the exitcode of a completed run. Otherwise, we cannot differentiate between a FAILURE happening due to SLURM-issues (e.g., transport failures), or a simply failing command. Otherwise, retrying would not work.
+
+2. Currently, the following parameters are passed to `srun` (calculated from the benchmark's parameters):
+   * `-t <hh:mm:ss>` CPU timelimit (generally, SLURM will round up to nearest minute)
+   * `-c <cpus>` number of cpus 
+   * `--threads-per-core=1` only use one thread per core
+   * `--mem-per-cpu <mem/cpus>` memory allocaiton in MBs per cpu
+   * `--ntasks=1` number of tasks per node
+
+3. The script parses the resulting job ID, and after the job finishes, runs `seff` to gather resource usage data:
+   * Exit code
+   * Status
+   * CPU time [s]
+   * Wall time [s]
+   * Memory [MB]
+
+## Limitations
+
+Currently, there are the following limitations compared to local benchexec:
+
+1. No advanced resource constraining / monitoring: only CPU time, CPU core and memory limits are handled, and only CPU time, wall time, and memory usage are monitored.
+2. No exotic paths in the command are handled: only the current working directory and its children are visible in the container
+3. The  user on the host and the container should not differ (due to using $HOME in the commands).
+4. Without singularity, no constraint is placed on the resulting files of the runs: this will populate the current directory with all the output files of all the runs.
+5. For timed-out runs, where SLURM terminated the run, no CPU time values are available.
+6. The executor only works with hyperthreading disabled, due to the inability to query nodes about the number of threads per core. Assuming it's always 2 is risky, as it may not hold true universally. Consequently, because we can only request whole cores from SLURM instead of threads, we must divide the requested number of threads by the threads-per-core value, which is unknown if hyperthreading could be enabled.
+7. Cancelling a benchmark run (by sending SIGINT) could be delayed up to a few minutes depending on the SLURM configuration.  

contrib/slurm/README.md

@@ -11,7 +11,10 @@ SPDX-License-Identifier: Apache-2.0
 -->
 # BenchExec Extension for Benchmarking via SLURM
 
-This Python script extends BenchExec, a benchmarking framework, to facilitate benchmarking via SLURM, optionally using a Singularity container.
+> [!IMPORTANT]  
+> The previous, single-job-based SLURM integration is no longer maintained. For its documentation, see [README-old.md](./README-old.md)
+
+This Python script extends BenchExec, a benchmarking framework, to facilitate benchmarking via SLURM array jobs using Singularity containers.
 
 In case of problems, please tag in an [issue](https://github.com/sosy-lab/benchexec/issues/new/choose): [Levente Bajczi](https://github.com/leventeBajczi) (@leventeBajczi).
 
@@ -23,69 +26,64 @@ In case of problems, please tag in an [issue](https://github.com/sosy-lab/benche
 ## Requirements
 
 * SLURM, tested with `slurm 22.05.7`, should work within `22.x.x`
-* Singularity (optional), tested with `singularity-ce version 4.0.1`, should work within `4.x.x`
+* Singularity, tested with `singularity-ce version 4.0.1`, should work within `4.x.x`
+* cgroup support is required 
 
 ## Usage
 1. Run the script with Python 3:
    ```
    python3 $BENCHEXEC_FOLDER/contrib/slurm-benchmark.py [options]
    ```
    Options:
-   - `--slurm`: Use SLURM to execute benchmarks. Will revert to regular (local) benchexec if not given. 
+   - `--slurm-array`: Use SLURM array jobs to execute benchmarks. Will revert to regular (local) benchexec if not given.
    - `--singularity <path_to_sif>`: Specify the path to the Singularity .sif file to use. See usage later.
    - `--scratchdir <path>`: Specify the directory for temporary files. The script will use this parameter to create temporary directories for file storage per-run, which get discarded later. By default, this is the CWD, which might result in temporary files being generated by the thousands in the working directory. On some systems, this must be on the same mount, or even under the same hierarchy as the current directory. Must exist, be writable, and be a directory.
    - `--retry-killed <N>`: Retry killed jobs (e.g., due to SLURM errors) this many times. Use -1 for unbounded retry attempts.
-   - `-N <N>`: Specify the factor of parallelism, i.e., how many instances to start at a time. Tested with up to `1000`, probably works with much higher values as well.
+   - `-N <N>`: Specify the factor of parallelism, i.e., how many jobs to submit at a time. Tested with up to `1000`, probably works with much higher values as well.
+   - `--aggregation-factor`: Put this many jobs into a single job of the array. 
+   - `--batch-size`: Allow this many runs inside a runcollection to be submitted. Lower values might hurt responsiveness, higher values might cause problems with script sizes. Suggested size is around a few thousand.
+   - `--parallelization`: Execute this many jobs in parallel inside a job of the array.
 
 ## Overview of the Workflow
 
-This works similarly to BenchExec, however, instead of delegating each run to `runexec`, it delegates to `srun` from SLURM.
+This works similarly to BenchExec, however, instead of delegating each run directly to `runexec`, it creates a hierarchy of run infos and an array job description for SLURM, which is then executed using `sbatch`. `runexec` is still used to measure and limit resources.
 
-1. If the `--singularity` option is given, the script wraps the command to run in a container. This is useful for dependency management (in most HPC environments, arbitrary package installations are frowned upon). For a simple container, use the following: 
+1. The script wraps the command to run in a container. This is useful for dependency management (in most HPC environments, arbitrary package installations are frowned upon). For a simple container, use the following: 
 
     ```singularity
-    BootStrap: docker
-    From: ubuntu:22.04
-
-    %post
-    apt -y update
-    apt -y install openjdk-17-jre-headless libgomp1 libmpfr-dev fuse-overlayfs
+      BootStrap: docker
+      From: ubuntu:24.04
+
+      %post
+      apt -y update
+      apt -y install <necessary packages for the tools>
+      apt -y install software-properties-common
+      add-apt-repository ppa:sosy-lab/benchmarking
+      apt -y install benchexec fuse-overlayfs
+      mkdir /work
+      mkdir /upper
     ```
 
-    Use `singularity build [--remote / --fakeroot] --fix-perms <name>.sif <name>.def` to build the container.
+    Use `singularity build [--remote / --fakeroot] --fix-perms <name>.sif <name>.def` to build the container. A remote service (e.g., [sylabs](https://cloud.sylabs.io/builder)) may be used if root permissions are missing.
 
-    Notice the `fuse-overlayfs` package. That is mandatory for the overlay filesystem to work properly.
+    Notice the `fuse-overlayfs` and `benchexec` packages. That is mandatory for the overlay filesystem to work properly and for `runexec` to exist in the container. 
 
     The script parameterizes `singularity exec` with the following params:
-    * `-B $PWD:/lower`: Bind the working directory to `/lower` (could be read-only)
+    * `-B "/sys/fs/cgroup:/sys/fs/cgroup"`: Bind the cgroup hierarchy for use inside the container
+    * `-B {basedir}`: Bind the "base directory" (directory of the .sif file) (can be read-only)
+    * `-B {workdir}:/lower`: Bind the current directory to `/lower` (can be read-only)
     * `--no-home`: Do not bind the home directory
     * `-B {tempdir}:/overlay`: Bind the temporary directory to `/overlay` (must be writeable)
-    * `--fusemount  "container:fuse-overlayfs -o lowerdir=/lower -o upperdir=/overlay/upper -o workdir=/overlay/work $HOME"`: mount an overlay filesystem at $HOME, where modifications go in the temp dir but files can be read from the current dir
-
-    We also wrap this command inside the container using `bash -c "{command} && echo 0 > exitcode || echo $? > exitcode` to save the exitcode of the process, _and_ always have 0 as the exitcode of a completed run. Otherwise, we cannot differentiate between a FAILURE happening due to SLURM-issues (e.g., transport failures), or a simply failing command. Otherwise, retrying would not work.
+    * `--fusemount  "container:fuse-overlayfs -o lowerdir=/lower -o upperdir=/overlay/upper -o workdir=/overlay/work {workdir}"`: mount an overlay filesystem at {workdir} under {basedir}, where modifications go in the temp dir
 
-2. Currently, the following parameters are passed to `srun` (calculated from the benchmark's parameters):
-   * `-t <hh:mm:ss>` CPU timelimit (generally, SLURM will round up to nearest minute)
-   * `-c <cpus>` number of cpus 
-   * `--threads-per-core=1` only use one thread per core
-   * `--mem-per-cpu <mem/cpus>` memory allocaiton in MBs per cpu
-   * `--ntasks=1` number of tasks per node
+2. A `--batch-size`-sized portion of the runs is organized into bins of size `--aggregation-factor`. Each bin will correspond to a job in the array. Inside each bin, `--parallelization`-many `runexec` instances can be started with exact resource allocations and usage reporting. Output files and output log are stored inside the temp dir. If an error is encountered (most commonly this is due to `fuse` locking up and causing a TIMEOUT without any logs being ready) the run is put into a second-chance queue to be run again, at most `--retry-killed` times. 
 
-3. The script parses the resulting job ID, and after the job finishes, runs `seff` to gather resource usage data:
-   * Exit code
-   * Status
-   * CPU time [s]
-   * Wall time [s]
-   * Memory [MB]
+3. The script parses the resource usage and status of each run, as it would with regular `runexec`. 
 
 ## Limitations
 
 Currently, there are the following limitations compared to local benchexec:
 
-1. No advanced resource constraining / monitoring: only CPU time, CPU core and memory limits are handled, and only CPU time, wall time, and memory usage are monitored.
-2. No exotic paths in the command are handled: only the current working directory and its children are visible in the container
-3. The  user on the host and the container should not differ (due to using $HOME in the commands).
-4. Without singularity, no constraint is placed on the resulting files of the runs: this will populate the current directory with all the output files of all the runs.
-5. For timed-out runs, where SLURM terminated the run, no CPU time values are available.
-6. The executor only works with hyperthreading disabled, due to the inability to query nodes about the number of threads per core. Assuming it's always 2 is risky, as it may not hold true universally. Consequently, because we can only request whole cores from SLURM instead of threads, we must divide the requested number of threads by the threads-per-core value, which is unknown if hyperthreading could be enabled.
-7. Cancelling a benchmark run (by sending SIGINT) could be delayed up to a few minutes depending on the SLURM configuration.  
+1. No exotic paths in the command are handled: only the directory of the `.sif` file and its children are visible in the container.
+1. The executor only works with hyperthreading disabled, due to the inability to query nodes about the number of threads per core. Assuming it's always 2 is risky, as it may not hold true universally. Consequently, because we can only request whole cores from SLURM instead of threads, we must divide the requested number of threads by the threads-per-core value, which is unknown if hyperthreading could be enabled.
+1. `fuse` sometimes locks up (more precisely: is in an uninterruptible state) for the entire duration of a job. My guess is the underlying lustre file system does not like it when the same path is overlayed from hundreds of nodes at the same time. As a mitigation, we re-run timed out jobs (not runs!).