Dispatcher can issue a migration request that comes back as failed even though it succeeded

[marusshi]

Let's say we want to migrate Foo from Pi0 to Pi1, using Dispatcher.

The current protocol is then:

1. Dispatcher sends pubsub message to Pi0 (migrate Foo)
2. Pi0 sends pubsub message to Pi1 (run my snapshot of Foo)
3. Foo successfully migrated to Pi1
4. Pi1 sends an ACK back to Pi0
5. Pi0 sends ACK back to Dispatcher (your request was successful)

If Pi0 dies after step 3 but before step 5, Dispatcher will never receive an ACK, the migrate request will timeout and the user will think the migration failed even though it succeeded.
Maybe instead of listening for an ACK from Pi0, should Dispatcher listen for ACK from Pi1?

[marusshi]

• Reason this came up is because I'm trying to handle some cases of a schedule failing to apply right now.
  Generally if Foo fails to migrate from Pi0 to Pi1, then we can usually assume Foo is not running on Pi0 or Pi, but then I ran into this issue.