pacemaker: remove node on delete (SOC-11240)

[sandonovsuse]

On node delete, pacemaker needs to remove the node from the cluster
prior to being deleted from crowbar. This change adds said feature.

[skazi0]

This loop will not work as expected. See https://stackoverflow.com/questions/31294630/is-it-safe-to-delete-from-an-array-inside-each for details.
TLDR
"when you delete an element, the elements following it will be shifted, hence the element that was supposed to be iterated next would be moved to the position of the deleted element, which has been iterated over already, and will be skipped."

[sandonovsuse]

I understand, but the iteration breaks after it is deleted, ie. no further iteration after unshift. I could do:

prole = roles.delete_at(index)
roles.unshift(prole)

though I already tested it, it should be OK. Test results are in bsc#1170479.

[skazi0]

There are actually more than one role with "pacemaker" in the name. I'm not sure what was the original intention here.
Maybe search for "pacemaker-cluster-member" if that's what you want.

[sandonovsuse]

Actually you're right, it should be pacemaker-config-$inst. Will make the changes

[jgrassler]

Hmm. So I tried to test this (with crowbar-core/crowbar-ha packages that contain this pull request and crowbar/crowbar-ha#376 as patches) and it doesn't yield what I'd expect to be the result. I deleted one controller...

root@crowbar:~ # knife node delete d52-54-77-77-01-02.vp5.cloud.suse.de d52-54-77-77-01-03.vp5.cloud.suse.de
Do you really want to delete d52-54-77-77-01-02.vp5.cloud.suse.de? (Y/N) y
Deleted node[d52-54-77-77-01-02.vp5.cloud.suse.de]
root@crowbar:

...and would have expected it to vanish from the stonith configuration in the cluster's proposal. It continues to be there though:

  "stonith": {
    "mode": "libvirt",
    "sbd": {
      "watchdog_module": "",
      "nodes": {
        "d52-54-77-77-01-01.vp5.cloud.suse.de": {
          "devices": [
            ""
          ]
        },
        "d52-54-77-77-01-02.vp5.cloud.suse.de": {  ### this should be gone if I understand the code correctly
          "devices": [
            ""
          ]
        },
        "d52-54-77-77-01-03.vp5.cloud.suse.de": {
          "devices": [
            ""
          ]
        }
      }
    },

[skazi0]

@jgrassler try to use "Forget" option in crowbar instead of direct deleting in chef.