Adds PROCESSORS property to tests based on number of ranks and threads

[kennyweiss]

This PR add the PROCESSORS property to tests in the blt_add_test macro to give CTest more context in scheduling its tests.

The number of processors is based on the number of mpi ranks -- provided through NUM_MPI_TASKS -- and the number of OpenMP threads -- provided through NUM_OMP_THREADS.

We also throw a FATAL_ERROR if the number of threads is not an integer or is less than 1.

Update:

• I tested this out in Axom (Improves test throughput, especially for threaded tests axom#1749) and the results are not the slam dunk I thought they were. Adding the PROCESSORS property to tests appears to slow the testing down somewhat (~3% for the Axom tests).
• So, we should probably hold off on merging this, or, at the very least, making it opt-in. I'm going to mark this PR as DRAFT.

[adayton1]

Thanks, @kennyweiss !

[white238]

Thanks @kennyweiss ! Did you notice a speed up after this?

[bartlettroscoe]

FYI: We have been using the PROCESSORS property for 15+ years with TriBITS/Trilinos and it works okay. Without this, CTest can't effectively schedule the running of tests that fill the cores but not overload them. You can sese how effective this is by going to:

• https://github.com/orgs/trilinos/discussions/15303#discussioncomment-17140029

and downloading and viewing the plots under the section "System RAM and Load Fracs Test Plot".

[aaroncblack]

@kennyweiss @white238 I tried adding PROCESSORS property on our tests in radar ( about 45 tests, mostly small, some multirank ), and it sped things up quite a bit for me. I tried running with 'ctest --parallel 112' on one of our nodes with 112 cores.

I'm adding the property myself after blt creates the test but maybe we can look at the behavior later. These tests aren't multi-threaded however.

One initial hiccup I had was I didn't make sure the srun jobs were launched with '--exclusive' so they were oversubscribed CPUs. (I was running ctest in an salloc).

After I added:

set(BLT_MPI_COMMAND_APPEND "--exclusive" CACHE STRING "")                                                                                                                                                                                        

to my cmake host config everything worked well.

I'll see if I can test your branch here.