Improve developer convenience around remote communication problems

[berndruecker]

We started to improve the developer experience for Spring users already by providing @ZeebeWorker(autoComplete=true) functionality. Now we want to go one step further and provide support for typical problems when communicating with Zeebe users have to solve on their own otherwise.

This is specifically about:

• Exceptions when completing jobs

  • There is a first implementation in DefaultCommandExceptionHandlingStrategy.java)
  • Retries in case of transient problems, but adding a delay (back-off), which is especially important if you get back pressure from Zeebe
  • Ignoring if a task was already completed, as this cannot be healed (Todo: Discuss if this should be known to the client code, as it might for example roll back its Spring transaction)
  • Fail otherwise, but decrement retries in Zeebe. This leads to Zeebe retrying the job, or an incident.

• Exceptions when starting process instances:

  • Retrying for connection problems, but only in a very short time frame, as the client code might be waiting. Or only allow for async client usage? Make sure retrying used back-off
  • How to deal with back pressure?

• Exceptions when correlating messages:

  • Retrying for connection problems, but only in a very short time frame, as the client code might be waiting? Make sure retrying used back-off
  • Can we get back pressure? How to deal with this?

[berndruecker]

@falko: Please comment if you have specific input, otherwise let's discuss again in our next meeting.

And just for visibility, commits so far: 5a609b7 & d41493c

[stephanpelikan]

We are supported a customer moving to C8. There was some kind of framework in place for C7 handling some tasks common to typical situations (for one aspect I also created another issue #426). One task is transaction handling - in case of C8 DB transactions in conjunction with eventual consistency.

Currently, we are trying ensure

1. business tasks done successfully are either committed as completed or rolled back
2. tasks cancelled (e.g. by boundary events) are rolled back

Ad 1:
In case of a successfully done task (e.g. service-task) it is important to ensure it is passed to Zeebe and not to repeat this action any more. At the moment a user of Spring Zeebe has to annotate the business method using @transactional. So if the business method is completed the transaction will be commited.
If there is an error connecting to Zeebe retry can be done (by DefaultCommandExceptionHandlingStrategy) but in case the client dies Zeebe will repeat the task since Zeebe only garantees "at least once" and not "exactly once". So, we need to store which task has already been done (e.g. based on the job-key which is hopefully to same for each attempt). This is similar to the outbox pattern but only to ensure idempotency. This record needs to be part of the same transaction so it has to be inserted by the business method what makes code verbose and less readable. It would be better, if the transaction is spawn by some kind of interceptor which can also hide the details of dealing with idempotency (in #426 we introduced such an interceptor to inject data into custom parameters of the business method).

Ad 2:
For our customer, who is from Telecom sector, there a lot of systems using asynchronous communication. It means service-tasks are not soley completed immediately, but also after a target system a responses to a request. Since this are technical aspects we don't want to have a send and a receive task instead of each service task. The same may happen for jobs doing synchronous workload but there a lot of them (e.g. multi-instance) what stretches total time of completion. Also in case the items cannot be passed to Zeebe for multi-instance the task takes a while to complete (4MB for Zeebe variables means only 104857 items if an item is a UUID what already hits amounts of bulk processing of certain business tasks in the end of the month). Another obvious example are user tasks.
However, as Zeebe's jobs are asynchronous we wanted to leverage this. Unfortunetly, if a task was cancelled, our business transaction is commited but the task cannot be completed by Spring Zeebe because it's gone. In this situation we would need to rollback the transaction but this is not possible because DefaultCommandExceptionHandlingStrategy handles this.
I already created some issue regarding this topic in Zeebe (camunda/camunda#11928) as a possible way to deal with this. Of course, there are many ways to handle this and I also heard you are thinking about kind of executionlisteners for Zeebe. In conjunction with 1 I don't see a way to handle cancelled tasks soley by Spring Zeebe.

Are there currently any things ongoing regarding this topic? Can you give some update?

[stephanpelikan]

By-the-way: Nice comment 😅
https://github.com/camunda-community-hub/spring-zeebe/blob/main/spring-client-zeebe/src/main/java/io/camunda/zeebe/spring/client/jobhandling/DefaultCommandExceptionHandlingStrategy.java#L55).

[stephanpelikan]

I also recently figured out another problem regarding transactions: On passing entities (e.g. JPA) as process-variables serialization of collections might fail due to lazy-loading. This is because the Hibernate session is not available on JSON serialization any more an the entity object is detached. To solve this one needs to manually set this to eager fetching. Downside is, that eager fetching is done in every use case and not only in those in which those values are passed to Zeebe as a process variable. If the transaction is created somewhere in Spring Zeebe then lazy loading still would work.