Skip to content

Introspection

The second reason this package exists. Signals structurally cannot answer any of these questions, and all of them are cheap once a registry and a table exist.

The catalogue

Every declared event, its payload shape, and everything listening to it - generated from the declarations rather than maintained beside them, so it cannot drift.

python manage.py export_catalogue                          # Markdown to stdout
python manage.py export_catalogue --format json
python manage.py export_catalogue --output docs/events.md
from django_domain_events import catalogue, render_catalogue

document = render_catalogue(catalogue(), format="json")

Both formats exist because they answer different questions. Markdown is read by a person onboarding onto a codebase. JSON is diffed by a pipeline that wants to fail a pull request for removing a field other teams consume:

python manage.py export_catalogue --format json --output events.json
git diff --exit-code events.json

Everything is sorted by name and every document ends in exactly one newline, so import order is not a difference and a checked-in catalogue produces a clean diff.

An event nothing listens to is the finding

The Markdown says "Nothing listens to this event." rather than leaving the section empty. That is the usual reason to read a catalogue at all.

Wildcard receivers, declared for AnyEvent, are listed once, in an Every event section at the top, and each event says "Plus every wildcard receiver" rather than repeating them - a transport listed under two hundred events is a catalogue nobody reads. An event with no receivers of its own says it still reaches the wildcards, because "nothing listens" would be false. A fan-out receiver has a line under its table naming the callable its targets come from, and the JSON carries the same as targets on each receiver and wildcard_receivers at the top level.

An event declared with a retention of its own says so in a line of prose under its heading - "Kept for 7 days", or "Deleted once every delivery has succeeded" - and the JSON carries it as retention_seconds and delete_when, the two values fire() records on the event row. An event on the default window has neither, and no line.

A receiver with a curve of its own or in a named lane gets a line under its table too - "retries on its own curve: base 120s, cap 1800s", "is served by relays started with --lane mail" - and the JSON carries backoff_base_seconds, backoff_cap_seconds and lane on every receiver. So does a declared give_up_after: "dead-letters a deferral once its delivery has been owed for 172800s", and give_up_after_seconds in the JSON, null where none is declared. A scheduled receiver says when it runs - "is due 172800s after the event occurred", or "is due at the time waitlist.events.offer_expiry returns for the event" - and the JSON carries due_seconds for a delay and due_callable for a callable, each null otherwise. Two fields rather than one that is sometimes a number and sometimes a path, so a JSON reader never switches on a value's type; a receiver with both null runs as soon as its event is recorded. Lines rather than columns, so the tables a project has already committed do not change shape for properties most receivers do not have. A half of the curve left to its setting is named as the setting rather than resolved, because the catalogue describes declarations and the machine that builds it need not share the relay's settings.

Building a catalogue never runs consumer code: a default_factory is named, not called.

Who listens to what

from django_domain_events import listens_for, what_listens_to

what_listens_to(OrderPlaced)  # [RegisteredReceiver, ...] sorted by key
listens_for("orders.reserve_stock")  # RegisteredEvent | None

what_listens_to spans every mode, not only the durable ones: "who reacts to this" is a question about the code, and an inline receiver is as much a reaction as a queued one.

It does not repeat wildcard receivers under each event. They receive it - they receive everything - and what_listens_to(AnyEvent) returns exactly them. listens_for returns None for a wildcard's key, since it is owed no single event; its delivery row already says which one it was.

listens_for is the direction an operator actually needs. A dead-letter row names a receiver key, and the next question is always what it was supposed to be receiving. None covers two cases that look identical from a delivery row - a key never declared, and one whose event class was deleted out from under it.

Quiet receivers

python manage.py quiet_receivers
python manage.py quiet_receivers --days 30
from datetime import timedelta

quiet_receivers(within=timedelta(days=30))

"This receiver has not received anything in ninety days" is a query here, not a guess.

Driven by the registry rather than by the table, so a receiver that has never received anything appears - which is the answer worth having, and exactly the one a query over delivery rows alone cannot produce, because there is no row to find.

Only DURABLE receivers are reported: the others leave no rows, so they have no history to be quiet about, and listing them as silent forever would train the reader to ignore the output.

Only a succeeded delivery counts. The question is whether the receiver did its work, not whether the relay tried - a row stuck failing for a month is exactly the case this must catch.

The answer survives the prune. Deleting delivery rows would otherwise delete the evidence that a receiver ran - within minutes, for an event deleted on consumption - so the prune records each receiver's newest success among the rows it deletes, in the same transaction as the delete, and this reports the later of that record and the rows still there. A receiver that never ran is still reported as never. The record is written only by the prune, never as deliveries succeed: one row per receiver, updated by every delivery, would queue a 20,000-row fan-out on a single lock.

The window defaults to RETENTION_DAYS, the window an ordinary event's deliveries are kept for.

The success time is read off succeeded_at, which is written on success and never cleared - unlike completed_at, which replay and requeue clear because a reopened row has not settled again. Reading the cleared column would tell an operator who had just replayed yesterday's events that the receiver they were re-running had never run at all.

Is the outbox keeping up?

python manage.py events_status
python manage.py events_status --format json
from django_domain_events import outbox_health

health = outbox_health()

quiet_receivers() answers whether a receiver is running; this answers whether the queue is draining. They fail differently, and both failures are real: a relay that has been down an hour has every receiver quiet and a backlog climbing, while one wedged receiver has a backlog and everything else fine.

Field Means
owed Deliveries due and not yet settled - pending, failed or claimed
scheduled Deliveries not yet settled whose due time is still ahead - a scheduled receiver waiting. Not counted as owed
claimed Currently leased by a worker
dead Dead-lettered. Not counted as owed
oldest_owed_at When the oldest still-owed delivery became owed: its due time where it has one, its event's recording otherwise; None when nothing is owed
lapsed_leases Claimed rows whose lease has expired
receivers Per-receiver backlog, worst first, omitting the ones with nothing owed, dead or scheduled. Each carries its own owed, scheduled, claimed, dead and oldest_owed_at; one with only scheduled work sorts after every receiver that is behind

A cancelled delivery is settled, so it appears in none of these: not owed, not scheduled and not dead. An outbox holding only cancelled work reads as drained.

The age of oldest_owed_at is the number to alert on, and it is measured from when the work arrived, not from when it next becomes available. That distinction is the whole point: the backoff pushes a failed delivery's available_at into the future on every attempt, so an alert written against that column reads a negative age exactly while a receiver is failing.

Work arrives when it becomes due. For most rows that is when the event was recorded; for a replayed or requeued row, when it was reopened; and for a scheduled receiver, its due time. A row due in a fortnight is scheduled until then and does not age at all, so a five-minute backlog alert does not fire for work that is on time - and one failing after it came due ages from its due time, not from an event a fortnight older.

It rises monotonically for as long as work sits undone - a relay that is down, a receiver that is failing its way through its retries, or a queue simply longer than the workers can drain.

It does not cover a receiver failing all the way to dead

A dead-lettered row leaves the owed set, so the backlog age drops back to None while nothing at all is being delivered. A dead letter is deliberately not owed - the relay will never pick it up again on its own, and counting it would leave a backlog alert firing forever after one bad deploy. Alert on dead as well. One number does not cover both.

Steady non-zero lapsed_leases means workers are dying mid-delivery, or a receiver outruns its lease and has its work thrown away every time - see lease_seconds. A relay that was asked to stop also gives its unstarted rows back by expiring their leases, so a brief non-zero reading after a deploy is that, not a dying worker.

The per-receiver claimed adds up to the total, and says whose rows the workers hold. A receiver with every owed row claimed and none finishing is wedged, or its task queue is sitting on the messages; the same total spread across receivers is a relay at work.

Every figure here is answered from an index and none of them scans the delivery table, so the call is cheap enough to scrape on an interval.

owed means "not terminal and due", a superset of what the relay can claim at any given moment: a row inside its backoff window is owed and not yet claimable. The superset is the useful side - this cannot report an empty queue while work is still outstanding. A row whose due time has not come is not outstanding yet, which is why it is scheduled instead; at exactly its due time it is owed, as the relay can claim it then. The prune settles by "not terminal", so it waits for owed and scheduled rows alike.

events_status prints the same figures, scheduled among them, and a line per receiver:

owed          3
scheduled     120
claimed       1
dead          0
lapsed leases 0
oldest owed   42s ago

  billing.send_invoice  owed=3  dead=0  scheduled=0 claimed=1   oldest=42s
  waitlist.expire_offer owed=0  dead=0  scheduled=120   claimed=0   oldest=-

--format json carries scheduled and claimed at the top level and on each receiver.

System checks

Run with python manage.py check.

id Level Fires when
E001 Error A receiver listens for a class that was never @event-decorated
E002 Error The configured CODEC cannot be imported
E005 Error A declared event has a field the configured codec cannot rebuild
E006 Error WAKE is neither "notify" nor "poll"
E007 Error NOTIFY_COALESCE_SECONDS is not a non-negative number
E008 Error RELAY_PRUNE is not a bool
E009 Error RELAY_PRUNE_SECONDS is not a positive number
E010 Error PRUNE_BATCH_ROWS is not a positive whole number
E011 Error AROUND_DELIVERY is not a string, does not import, or names something not callable
W001 Warning Deliveries are owed to a receiver key the registry no longer has
W002 Warning Deliveries are owed for an event name the registry no longer has
W006 Warning A settings dict is named DOMAIN_EVENTS rather than DJANGO_DOMAIN_EVENTS
W007 Warning DJANGO_DOMAIN_EVENTS holds a key this package never reads

E001 matters because nothing else would ever say so: the event cannot be fired, so there is no failure to observe, only silence.

E002 runs at startup because a codec is imported lazily - without it, the first symptom of a missing extra is a delivery failing in a worker.

E005 asks the harder half of the same question. E002 says the codec can be imported; E005 says it can decode the events this project declares, which is what actually fails. The asymmetry is what makes it worth a check: encoding accepts whatever the annotation says and commits it inside the caller's transaction, so the event exists, and only the decode refuses - in the relay, in another process, one dead letter per receiver. A codec that does not implement unsupported_fields is not interrogated, because this package should not guess at what a codec it did not write can do.

W006 and W007 exist because both mistakes are otherwise silent. setting() reads only the keys this package asks for, so a typo sits in the settings looking effective; and the dict is DJANGO_DOMAIN_EVENTS while the app label and most prose say "domain events", so a DOMAIN_EVENTS block returns nothing and every value falls back to its default with nothing said.

W002 catches a renamed event, which W001 cannot see: the receivers keep their keys, so nothing looks orphaned, while every row written under the old name now decodes to nothing and spends one attempt budget at a time finding out. Fix it by pinning the old identity with @event(name=...) on the class that replaced it.

Both warnings are limited to work still owed, and both mean the same thing by it: any status the relay would still claim - which includes claimed, since a worker that dies leaves the row claimed under a lapsed lease. A settled row naming a retired event is history, and warning about history on every check run teaches the reader to skip the output.

The admin

Add django.contrib.admin and both models appear, read-only, with actions.

  • Event records - filter by name and date, see how many deliveries are still owed per event, Replay selected events, and Cancel owed deliveries of selected events.
  • Delivery records - the dead-letter queue, filterable by status and receiver - cancelled included, so withdrawn work can be listed on its own - with Requeue selected dead deliveries and Cancel selected deliveries. Each row shows its target, and the search covers the receiver key, the target and the last error, since the target is what an operator is holding when one destination of a fan-out says it never received something.

Both are read-only, and that is deliberate: the one guarantee this package sells is that a row exists if and only if the change committed, and a form that can write one is a way to break it. Editing status by hand is how a claimed row gets handed to a second worker.

Deleting is refused too. It would cascade owed deliveries away with no record that anything was lost; prune_events is the supported route, because it re-checks settledness at the delete itself.

The requeue action goes through requeue_dead() rather than a bulk update, so it resets the attempt budget, clears the stale lease and error, and wakes the relay. It reports what it skipped: a mixed selection requeues only the dead rows.

Both cancel actions go through cancel_deliveries(), so a row a worker claimed after the page loaded is left to run rather than settled under it, and a cancel issued while a replay of the same event is open waits for it. On the delivery changelist the selected rows are what is withdrawn and nothing else - a sibling receiver's row of the same event stays owed - and the message counts the rows cancelled, the ones in flight, and the ones already settled. On the event changelist it is every owed row of each selected event, reported as cancelled and in flight.

Every action needs the model's change permission

Django offers an action with no declared permission to anyone who can reach the changelist, and has_change_permission gates the form alone. Replay re-runs every durable receiver - re-sent emails, re-called webhooks, and a fan-out receiver's targets asked for again - and a cancel withdraws work the application is owed, so a view-only grant must carry neither. Give change_eventrecord / change_deliveryrecord to whoever may run them; the edit form stays refused either way.

The filters are built from the registry, not from the table: a SELECT DISTINCT name over an event log is a full scan on every page load, and an event declared but never fired would not appear. Selecting one and seeing an empty list is itself the finding.