Recallbacks

[MTCam]

Continued from PR (#370):

This builds on @thomasgibson callbacks and uses the opportunity to clean up some cumbersome junk in the infrastructure that was gumming up the use of the code in production. Instead of building the callback and health checking facilities on old junky code that we didn't like, we refactored it a bit here, to give the user more flexibility, and responsibility, for the behavior of the simulation application.

• Updates the old callback (checkpoint) to the explicitly named (pre_step_callback)
• Takes apart the clunky sim_checkpoint, sim_vizualization, for a more direct, much less "fluid-specific" interface
• Allows full customization of status messages, viz fields, and health checking
• Implements minimally collective health-check (only one collective comm to sync)
• Optionally dumps viz at health fail (automatically collective)
• Correctly places control over what to do with errors in the hands of the user/driver

[MTCam]

Ready for another peak. @majosm and @thomasgibson.

[majosm]

Looking good! I left a few minor comments.

[MTCam]

I've built-in a health failure to autoignition example, so CI should fail that example. Don't be alarmed :)

[thomasgibson]

I assume this was left-over testing that can be removed?

[MTCam]

Yes - just wanting to demonstrate the health error while we have this PR cracked open. I can certainly remove it right away if it is distracting.

[thomasgibson]

Think it's worth adding in a utility function in simutils that does this reduction across ranks? Just to clean this up. I think @majosm has some code somewhere that does this. This is not a functional request, just aesthetics.

[MTCam]

health_errors = allsync(my_health_check(dv), op=MPI.LOR) ?
Oh woops, we'd need to pass in comm too. Hrm...

[thomasgibson]

I mean, comm is already being used around so is it that bad pass it? I don't see the problem.

[majosm]

Here is the code snippet @thomasgibson was referring to:

mirgecom/mirgecom/mpi.py

Lines 161 to 166 in e47d3d0

	def comm_any(comm, local_value):
	"""Return True if *local_value* is True on any rank in *comm*."""
	reduced_value = np.array([int(local_value)])
	from mpi4py import MPI
	comm.Allreduce(MPI.IN_PLACE, reduced_value, op=MPI.MAX)
	return bool(reduced_value[0])

(Don't copy the implementation, it's dumb. Just use MPI.LOR.)

Edit: I'm not sure this adds anything as-is. You might be able to stick the if comm is not None: check in here though.

[MTCam]

Not a problem at all!

[MTCam]

Just real quick before I implement..... @thomasgibson and @majosm

How would we feel about this utility:

def allsync(local_value(s), comm, op=MPI.MAX):
   if comm is None:
     return local_value(s)
   return comm.allreduce(local_values, op=op)
 
global_value(s) = allsync(local_value(s), comm, op=MPIOP)

This way the user can use the allsync utility to do syncing operations on arrays of data, or other objects as supported by comm, and control over what operation is performed.

[thomasgibson]

@MTCam I just left some minor comments, but overall I'm happy with the auto-ignition example. I think once the changes are applied to the other examples, we can finally merge this.

[MTCam]

I think this is ready for your eyes again @thomasgibson and @majosm. The only changes were the ones we discussed (5306e30), (5bd30d8).

[majosm]

Looks good to me. 👍