Skip to main content

Task lifecycle and retries

Supplier lifecycle​

A Supplier, when started, does the following:

  • analyses capabilities of Generators and Orchestrators and builds a manifest
  • registers with Sixpack by sending the manifest
  • listens for incoming tasks from Sixpack

The SDK does this automatically. The developer mainly implements Generators and Orchestrators.

Registration​

Registration updates the Sixpack catalogue:

  • the Supplier's Generators and Orchestrators become available in the catalogue
  • if some datasets must be stocked, Sixpack starts creating them
  • if an item definition changes, Sixpack evaluates whether existing stocked datasets are still compatible

During registration, Sixpack also checks SDK compatibility:

  • deprecated SDK versions print warnings in the log
  • incompatible SDK versions prevent the Supplier from starting

Task handling​

Sixpack sends tasks to the Supplier. The Supplier routes each task to the right Generator or Orchestrator method.

Sixpack retries every task several times before considering the task permanently failed. This applies to both Generators and Orchestrators. Because Orchestrators may call multiple child items, retry patterns can become nested, but Sixpack manages that automatically.

For the exception choices that influence retry behaviour, see Exception handling.

State handling​

Because of retries and distributed execution, state management is important.

Generators​

Sixpack treats Generator methods as idempotent. If a Supplier is killed while a Generator task is in progress, the task can be restarted when the Supplier becomes available again, provided that the Generator is still defined.

Generators may have stateful components, such as database connections. The Generator is responsible for managing those components. Sixpack does not manage target-system state for the Generator.

Orchestrators​

Orchestrators are stateless by design and can be killed at any time. Sixpack continues execution where it left off after the Supplier restarts, provided that the Orchestrator is still defined.

If Orchestrator code changes structurally while a dataset is in progress, that dataset may fail. The impact depends on how far the execution has progressed compared with the changed code path.

Practical retry rules​

  • Develop Generators as idempotent operations.
  • When using random values, verify in target systems that uniqueness constraints are not violated.
  • When uniqueness or other fatal constraints are violated, throw a non-retryable exception.