fix(events): fail closed when the enqueue fails

[vincent-pochet]

Part of ING-606

Context

Events::CreateService commits the event row and only then enqueues Events::PostProcessJob. The two are not transactional, so when Redis is down the row is committed and the job never runs: no pay-in-advance fee is created, and reposting the same transaction_id answers value_already_exist forever. Events::CreateBatchService has the same shape, and one bulk push failure strands the whole batch.

Description

• Remove the event, or the whole batch, when its post-processing cannot be enqueued, so the caller's retry succeeds instead of being refused as a duplicate.
• Move the enqueue ahead of the Kafka produce, so a failed enqueue publishes nothing downstream.
• Scope the ArgumentError rescue to the timestamp parse: ActiveJob::SerializationError is an ArgumentError, so an enqueue failure was reported as an invalid timestamp.

Hard-deleted rather than discarded: index_unique_transaction_id has no deleted_at predicate, so a discarded event keeps refusing the retry. Enqueuing inside the transaction is worse: the job is pushed before the commit, so every event would race its own commit instead of only the rare accepted-then-failed enqueue.

Behavior change to be aware of

• Aggregation reads events directly, so a stranded event still billed pay-in-arrears charges correctly. Rejecting it drops that usage unless the caller retries.
• Narrows the window rather than closing it: a process killed between commit and cleanup still strands the event.

[lago-claude-ai-agent]

Automated pre-review (advisory, not a required check) — verdict: HOLD · CI green

HOLD — The enqueue-failure cleanup hard-deletes Event records via delete/delete_all, despite Event including Discard::Model and the repository rule prohibiting hard deletion of soft-deletable models. Rework both paths without bypassing the model’s deletion policy while preserving retryability, and make the specs prove the retry/physical-row behavior (Event.count is default-scoped and would also pass after a discard).

[lago-claude-ai-agent]

Automated pre-review (advisory, not a required check) — verdict: HOLD · CI green

HOLD — the enqueue-failure cleanup hard-deletes Event, despite Event being soft-deletable and the repository explicitly forbidding hard deletion of soft-deletable models. Rework the cleanup/idempotency design through the supported model lifecycle and a dedicated service; the non-partial uniqueness constraint means simply substituting discard is not sufficient.

Add direct job coverage proving that a queued Events::PostProcessJob whose event has disappeared is discarded without invoking post-processing; the new discard_on behavior is currently untested.

[lago-claude-ai-agent]

Automated pre-review (advisory, not a required check) — verdict: HOLD · CI green

HOLD — both enqueue-failure handlers hard-delete Event, even though it includes Discard::Model; this bypasses the repository’s soft-deletion contract.

• Rework the single and batch rollback paths so retryability is preserved without event.delete/delete_all. Also cover an enqueue accepted before an adapter error: the current immediate-raise specs do not catch a queued job left referencing a deleted event.

[lago-claude-ai-agent]

Automated pre-review (advisory, not a required check) — verdict: PASS · CI green

PASS — The transaction/enqueue ordering is consistent across both PostgreSQL ingestion paths while preserving ClickHouse behavior. The focused specs cover rollback and retryability on enqueue failure, Kafka ordering, serialization errors, and the pre-commit deserialization race.

[toommz]

Nice catch 👏