Update pe layouts alphabranch

[mnlevy1981]

Description of changes

Same as #424 but to alphabranch instead of master (cherry-picked the two commits I made on the previous branch)

[billsacks]

There was some discussion in #424 that I'm copying here for future reference:

From myself:

If I remember correctly (which I may not be!), for CESM2, we made the default PE layout significantly smaller than the layout used in the cmip runs. I feel like that's a good idea for the sake of testing. The ideal in my mind is to have the default layout be smaller, and then also define a "large" (or something like that) PE layout if that option is given to create_newcase.

@mnlevy1981 @fischer-ncar @briandobbins - thoughts on this?

From @briandobbins

My idea, though I'd greatly appreciate feedback here, is to have the 'default' (the M size) indeed be smaller than the target CMIP layout.

Since most costs are dictated by the atmosphere, especially with the MTt4s runs, I was thinking of using:

X (or X1) = CAM has 5400 MPI ranks (1 element/core) each with 2-3 vertical/tracer threads (lots of nodes!)
L = CAM has 5400 MPI ranks (1 element/core) but no threading
M = CAM has roughly 1350 - 2700 MPI ranks (2-4 elements/ core) - default, and efficient, but not fast
S = CAM has minimal number of nodes that can run the problem and fit in memory - most efficient, but very slow.

(Typically, CAM-SE gets the best performance when all ranks have the same number of elements -- a decomposition where some ranks have 1 element and others have 2 will be performance-gated by the ones with 2.)

This would require a '--pecount L (or X/X1)' option during case creation for the CMIP runs, but otherwise be more efficient for non-CMIP users. I did look but didn't see a 'PT' option for case creation, Bill, that you mentioned the other day, but also didn't have much time to look. I'll check again when I get a moment.

Thoughts?

From @mnlevy1981

@billsacks As @briandobbins mentioned, I don't think this should be the default layout for the release... but my concern is that right now the layout that comes out of the box tries to squeeze the whole model onto like 6 nodes and the model doesn't run in that configuration. I figured there is currently only one use-case for this compset, and that's the CESM3 development runs - so let's provide the layout being used in those runs (currently by copying an env_run.xml file into every case root) and then we at least have something reasonable out of the box while we think about what other layouts we want to provide

From myself

@briandobbins - I like your suggestion.

I looked around briefly and I'm not sure if we still support _PT. I found ESMCI/cime#900, where a comment from Jim Edwards said we dropped that, then a later comment said that it's working. From a quick look through the code and some xml files, my sense is that it might actually support any arbitrary string. That said, I think I actually like your suggestion of using "X" here rather than "T", since I think it gives a better indication of what's going on - it's an even larger layout, and the user doesn't really need to know if it happens to use threads to achieve that.

I see your point, @mnlevy1981 . I can see things both ways and I guess I don't have strong feelings... but I imagine that for testing (in system tests or manual tests) someone would want to use a substantially smaller layout.