Task Lifecycle

Asynchronous tasks go through multiple states during their lifecycle. This page records the lifecycle of a task from creation to deletion.

When you enqueue a task, asynq manages the task internally to ensure that the handler is called at the specified time. During this process, the task may go through different lifecycle states.

Here is a list of different lifecycle states:

  • Scheduled: The task is waiting for future processing (only applicable to tasks with ProcessAt or ProcessIn options).
  • Pending: The task is ready for processing and will be picked up by an idle worker.
  • Active: The task is being processed by a worker (i.e., the handler is processing the task).
  • Retry: The worker is unable to process the task, and the task is waiting for a future retry.
  • Archived: The task has reached the maximum retry attempts and is stored in the archive for manual inspection.
  • Completed: The task has been successfully processed and will be retained until the retention time expires (only applicable to tasks with the Retention option).

Let's look at the different lifecycle states with an example.

// Task 1: Scheduled for processing in 24 hours.
client.Enqueue(task1, asynq.ProcessIn(24*time.Hour))

// Task 2: Enqueued for immediate processing.
client.Enqueue(task2)

// Task 3: Enqueued with retention option.
client.Enqueue(task3, asynq.Retention(2*time.Hour))

In this example, Task 1 will stay in the Scheduled state for the next 24 hours. After 24 hours, it will transition to the Pending state and then to the Active state. If the task is successfully processed, the task data will be deleted from Redis. If the task is not processed successfully (i.e., the handler returns an error or panics), the task will transition to the Retry state for later retries. After a period of time, the task will transition back to the Pending state and then to the Active state. This cycle will continue until the task is successfully processed or exhausts the retry attempts. In the latter case, the task will transition to the Archived state.

In this example, the only difference between Task 2 and Task 1 is that Task 2 skips the Scheduled state and directly enters the Pending state.

Task 3 is enqueued with a retention option of 2 hours. This means that after a worker successfully processes Task 3, the task will remain in the Completed state for 2 hours before being removed from the queue. By default, if a task does not have a retention option set, it will be removed from the queue immediately after completion.

The diagram below illustrates the state transitions.

+-------------+            +--------------+          +--------------+           +-------------+
|             |            |              |          |              | Success   |             |
|  Scheduled  |----------->|   Pending    |--------->|    Active    |---------> |  Completed  |
|  (Optional) |            |              |          |              |           |  (Optional) |
+-------------+            +--------------+          +--------------+           +-------------+
                                  ^                       |                            |
                                  |                       |                            | Deletion
                                  |                       | Failed                     |
                                  |                       |                            V
                                  |                       |
                                  |                       |
                           +------+-------+               |        +--------------+
                           |              |               |        |              |
                           |    Retry     |<--------------+------->|   Archived   |
                           |              |                        |              |
                           +--------------+                        +--------------+