Introduction

Last edited 4 minutes ago.

On this page

A backend in Apalis is the bridge between your application and a task store. It is responsible for delivering tasks to workers, signalling liveness, providing middleware hooks, and encoding task data for storage. Any queue technology — Redis, PostgreSQL, SQLite, in-memory — becomes an Apalis backend by implementing two core traits: Backend and BackendExt.

In other ecosystems you may know this concept as a broker, queue, or transport. In Apalis, the backend abstraction is deliberately generic so the same worker logic runs against any storage layer.


The Backend Trait

Backend defines the runtime contract between a worker and a task store. It covers three responsibilities:

  1. Polling — producing a stream of tasks for a worker to execute
  2. Heartbeating — emitting periodic liveness signals so the system can detect stalled workers
  3. Middleware — exposing a composable layer for cross-cutting concerns (retries, tracing, rate limiting, etc.)
pub trait Backend {
    /// The type of job arguments this backend delivers to workers.
    type Args;

    /// The type used to uniquely identify each task (e.g. `Uuid`, `i64`).
    type IdType: Clone;

    /// Arbitrary context attached to every task — for example, priority,
    /// scheduled time, or retry count. Must implement `Default` so the
    /// backend can construct context for new tasks.
    type Context: Default;

    /// The error type returned by backend operations such as polling
    /// or heartbeating.
    type Error;

    /// The stream of tasks returned by `poll`. Each item is:
    /// - `Ok(Some(task))` — a task is available for processing
    /// - `Ok(None)`       — the backend is idle (no tasks right now)
    /// - `Err(e)`         — a backend error occurred
    type Stream: Stream
        Item = Result<Option<Task<Self::Args, Self::Context, Self::IdType>>, Self::Error>,
    >;

    /// The stream returned by `heartbeat`. Each `Ok(())` item represents
    /// a successful liveness tick. An `Err` signals that the heartbeat
    /// mechanism has failed.
    type Beat: Stream<Item = Result<(), Self::Error>>;

    /// The middleware layer type for this backend. Used by the Apalis
    /// service builder to compose cross-cutting behaviour (e.g. retries,
    /// timeout, tracing) around task execution.
    type Layer;

    /// Produces a heartbeat stream for the given worker. The worker
    /// drives this stream to periodically register its liveness with
    /// the backend, enabling dead-worker detection and task requeuing.
    fn heartbeat(&self, worker: &WorkerContext) -> Self::Beat;

    /// Returns the backend's middleware layer. This is called once
    /// during worker setup and composed into the service stack.
    fn middleware(&self) -> Self::Layer;

    /// Begins polling the backend for tasks assigned to this worker,
    /// returning a decoded task stream. For most use cases you will
    /// drive this stream inside a [`Worker`].
    ///
    /// If you need access to the raw encoded representation, see
    /// [`BackendExt::poll_compact`].
    fn poll(self, worker: &WorkerContext) -> Self::Stream;
}

Associated Types at a Glance

TypePurpose
ArgsThe decoded job argument type your handler receives
IdTypeUnique task identifier — often Uuid or a database row ID
ContextPer-task metadata (priority, attempt count, scheduled time, …)
ErrorUnified error type for all backend I/O operations
StreamAsync stream of decoded tasks for a worker
BeatAsync stream of heartbeat ticks
LayerMiddleware type composed into the worker's service stack

The BackendExt Trait

BackendExt extends [Backend] with serialization awareness. While Backend works in terms of fully decoded Args, BackendExt exposes the compact (encoded) representation that the backend actually stores. This separation keeps storage concerns out of your handler logic while still allowing low-level access when needed.

pub trait BackendExt: Backend {
    /// The codec used to encode `Args` into `Compact` for storage and
    /// decode them back again on the worker side. See [`Codec`] for details.
    type Codec: Codec<Self::Args, Compact = Self::Compact>;

    /// The encoded representation of task arguments as stored in the
    /// backend — for example, a `Vec<u8>` for binary formats or a
    /// `serde_json::Value` for JSON-based stores.
    type Compact;

    /// Like `Backend::Stream`, but yields tasks in their *encoded* form.
    /// Useful for inspecting or forwarding tasks without a full decode
    /// round-trip, or for implementing custom deserialization paths.
    type CompactStream: Stream
        Item = Result<Option<Task<Self::Compact, Self::Context, Self::IdType>>, Self::Error>,
    >;

    /// Returns the [`Queue`] identifier associated with this backend
    /// instance — typically the queue name or topic the backend is
    /// bound to.
    fn get_queue(&self) -> Queue;

    /// Like `Backend::poll`, but yields tasks in their compact (encoded)
    /// form without deserializing `Args`. Use this when you need direct
    /// access to the raw payload — for example, when routing tasks to
    /// different workers based on the encoded content.
    fn poll_compact(self, worker: &WorkerContext) -> Self::CompactStream;
}

Associated Types at a Glance

TypePurpose
CodecHandles encoding Args → Compact and decoding Compact → Args
CompactThe storage-level representation (e.g. Vec<u8>, String)
CompactStreamAsync stream of encoded tasks — bypasses deserialization

How Backend and BackendExt Relate

           ┌─────────────────────────────────────────┐
           │              BackendExt                 │
           │  ┌───────────────────────────────────┐  │
           │  │             Backend               │  │
           │  │  heartbeat()  middleware()  poll() │  │
           │  └───────────────────────────────────┘  │
           │  get_queue()  poll_compact()  Codec      │
           └─────────────────────────────────────────┘
  • Backend is the minimum required interface for a working job queue. All workers depend only on Backend.
  • BackendExt adds serialization details needed by [TaskSink] (for pushing tasks) and advanced consumers that need raw payloads.
  • A type that implements BackendExt automatically satisfies Backend, since BackendExt: Backend.

Implementing a Custom Backend

Any type can become an Apalis backend. At minimum you implement Backend; add BackendExt when the type also needs to act as a [TaskSink] target or expose its codec.

use apalis::prelude::*;

struct MyBackend { /* ... */ }

impl Backend for MyBackend {
    type Args    = MyJob;
    type IdType  = uuid::Uuid;
    type Context = MyJobContext;
    type Error   = MyBackendError;
    type Stream  = /* ... */;
    type Beat    = /* ... */;
    type Layer   = /* ... */;

    fn heartbeat(&self, worker: &WorkerContext) -> Self::Beat { /* ... */ }
    fn middleware(&self) -> Self::Layer { /* ... */ }
    fn poll(self, worker: &WorkerContext) -> Self::Stream { /* ... */ }
}

For most use cases, you will use one of Apalis's built-in backends rather than writing your own. See the Backends overview for the full list.


Implementations

BackendCrateBest For
PostgreSQLapalis-postgresDurable jobs, existing Postgres infra
MySQL/MariaDBapalis-mysqlDurable jobs, existing MySQL infra
SQLiteapalis-sqliteLow-traffic or single-node deployments
Redisapalis-redisHigh-throughput, low-latency job queues
AMQPapalis-amqpMessage broker-based architectures
PGMQapalis-pgmqPostgres-native message queues with at-least-once delivery
NATSapalis-natsDistributed messaging, cloud-native and edge deployments
RSMQapalis-rsmqRedis-backed simple message queues
Cronapalis-cronSchedule-driven jobs — use with pipe_to for persistence

Summary

TraitResponsibility
BackendTask polling, heartbeating, middleware composition
BackendExtCodec configuration, queue identity, encoded task access

Together these two traits form a clean boundary between how tasks are stored and retrieved and how they are processed — letting you swap backends without touching your job handler logic.