[Bug]: Pushing a fix over a crashlooping app never replaces the running droplet; CLI gets an opaque 500

[norman-abramovitz]

What happened?

With an app whose current droplet crashloops, pushing a corrected package does not recover the app. Staging succeeds, the new droplet is created and set as current, and the app workload StatefulSet spec is updated to the new droplet image — but the crashlooping pod is never replaced, so the old droplet keeps running (and crashing) indefinitely.

The mechanism appears to be a known Kubernetes StatefulSet behavior: a RollingUpdate never deletes a pod that has not become Ready, so a crashlooping pod blocks its own replacement. Korifi's set-current-droplet handler then times out awaiting the app state:

"logger":"handlers.app.set-current-droplet" ... "error":"failed to await the app staged condition:
object <ns>/<app-guid> state has not been met in 120.000000s"

and the CFApp reconciler loops on:

"message":"Reconciler error" ... "error":"resource not ready: DesiredStateNotReached"

The CLI surfaces all of this as an opaque 500:

Unexpected Response
Response Code: 500
Code: 0, Title: , Detail: {"errors":[{"detail":"An unknown error occurred.","title":"UnknownError","code":10001}]}

Observed state after the "fix" push: CF reports the new droplet as current, the StatefulSet spec has the new droplet image, but the pod still runs the old image in CrashLoopBackOff. Repeat pushes do not help. cf restart (a full stop/start, which scales the StatefulSet down and up) does recover the app onto the new droplet — but nothing tells the user that, and the push they just ran reported failure despite having half-succeeded.

The initial push of the crashing app fails with the same opaque 500 (the start wait cannot converge), which is arguably expected — but the error detail gives the user nothing to go on in either case.

What you expected to happen

Pushing a fixed package over a crashed/crashlooping app replaces it and starts the new droplet, as on classic CF — without requiring the user to know to run cf restart afterwards. Secondarily, when the handler does fail, surfacing the underlying reason (await timeout / DesiredStateNotReached) instead of UnknownError/10001 would make this diagnosable from the CLI.

Acceptance Criteria

GIVEN an app whose current droplet crashloops
WHEN I cf push a corrected package for that app
THEN the app runs the new droplet without additional manual intervention

GIVEN the set-current-droplet handler fails an await
WHEN the error is returned to the CLI
THEN the response identifies the cause rather than returning UnknownError/10001

How to reproduce it (as minimally and precisely as possible)

App dir containing only a Procfile, plus this manifest:

applications:
  - name: crashrepro
    memory: 128M
    no-route: true
    health-check-type: process
    buildpacks:
      - paketo-buildpacks/procfile

1. Procfile = web: /bin/false; cf push crashrepro. Push fails with the 500 above; app ends up in CrashLoopBackOff. (So far arguably expected.)
2. Change Procfile to web: sleep infinity; cf push crashrepro again.
3. Staging succeeds and the new droplet becomes current (cf curl /v3/apps/:guid/droplets/current), the StatefulSet spec shows the new droplet image, but the pod keeps crashlooping on the old image. The push reports the same 500. Repeat pushes never recover the app.
4. cf restart crashrepro recovers it — the app starts on the new droplet.

Anything else we need to know?

Possibly related to #3234 (set-current-droplet semantics vs classic CF's revisions model) — same area, different symptom: here the droplet swap is accepted but never lands on the workload. Checked commits on main since v0.18.0 and found nothing touching this path.

Environment

• Revision of codebase: Korifi v0.18.0 (helm chart korifi-0.18.0, kind installer)
• Kubernetes version (use kubectl version): v1.35.0 (kind)
• Cloud provider or hardware configuration: kind on Docker Desktop, macOS/arm64 host
• Others: cf CLI 8.18.3