Dramatiq is a simple task queue implementation for Python3. dramatiq-pg provides a Postgres-based implementation of a dramatiq broker.

This is a fork of the original dramatiq-pg project, updated to support Dramatiq 2.0+ and Python 3.10+.

• Updated dramatiq dependency from ^1.5 to ^2.0
• Updated minimum Python version from 3.6 to 3.10
• Added testcontainers for running functional tests with Docker (no local PostgreSQL required)

• Super simple deployment: Single table, no ORM.
• Stores message payload and results as native JSONb.
• Uses LISTEN/NOTIFY to keep worker sync. No polling.
• Implements delayed task.
• Reliable thanks to Postgres MVCC.
• Self-healing: automatic purge of old messages. Automatic recovery after crash.
• Utility CLI for maintainance: flush, purge, stats, etc.

Note that dramatiq assumes tasks are idempotent. This broker makes the same assumptions for recovering after a crash.

• Install dramatiq-pg package from PyPI:

  $ pip install dramatiq-pg psycopg2-binary

  Ensure you have either psycopg2 or psycopg2-binary installed.
• Init database schema with init command.

  $ dramatiq-pg init

  Or adapt dramatiq-pg/schema.sql to your needs.
• Before importing actors, define global broker with a connection pool:

  import dramatiq
  import psycopg2.pool
  from dramatiq_pg import PostgresBroker
  
  dramatiq.set_broker(PostgresBroker(i))
  
  @dramatiq.actor
  def myactor():
      ...

Now declare/import actors and manage worker just like any dramatiq setup. An example script is available, tested on CI.

The CLI tool dramatiq-pg allows you to requeue messages, purge old messages and show stats on the queue. See --help for details.

Dramatiq-pg documentation is hosted on GitLab and give you more details on deployment and operation of Postgres as a Dramatiq broker.

Django : Use django-dramatiq-pg by Curtis Maloney. It includes configuration, ORM model and database migration.

If you encounter a bug or miss a feature, please open an issue on GitLab with as much information as possible.

dramatiq_pg is available under the PostgreSQL licence.

Thanks to all contributors :

• Andy Freeland
• Curtis Maloney, Django support.
• Federico Caselli, bugfixes.
• Giuseppe Papallo, bugfixes.
• Rafal Kwasny, improvements.

The logo is a creation of Damien CAZEILS