Introspection¶
The second reason this package exists. Signals structurally cannot answer any of these questions, and all of them are cheap once a registry and a table exist.
The catalogue¶
Every declared event, its payload shape, and everything listening to it - generated from the declarations rather than maintained beside them, so it cannot drift.
python manage.py export_catalogue # Markdown to stdout
python manage.py export_catalogue --format json
python manage.py export_catalogue --output docs/events.md
from django_domain_events import catalogue, render_catalogue
document = render_catalogue(catalogue(), format="json")
Both formats exist because they answer different questions. Markdown is read by a person onboarding onto a codebase. JSON is diffed by a pipeline that wants to fail a pull request for removing a field other teams consume:
python manage.py export_catalogue --format json --output events.json
git diff --exit-code events.json
Everything is sorted by name and every document ends in exactly one newline, so import order is not a difference and a checked-in catalogue produces a clean diff.
An event nothing listens to is the finding
The Markdown says "Nothing listens to this event." rather than leaving the section empty. That is the usual reason to read a catalogue at all.
Wildcard receivers, declared for AnyEvent, are listed once, in an Every
event section at the top, and each event says "Plus every wildcard receiver"
rather than repeating them - a transport listed under two hundred events is a
catalogue nobody reads. An event with no receivers of its own says it still
reaches the wildcards, because "nothing listens" would be false. A fan-out
receiver has a line under its table naming the callable its targets come from,
and the JSON carries the same as targets on each receiver and
wildcard_receivers at the top level.
An event declared with a retention of its own says so in a line of
prose under its heading - "Kept for 7 days", or "Deleted once every delivery has
succeeded" - and the JSON carries it as retention_seconds and delete_when,
the two values fire() records on the event row. An event on the default window
has neither, and no line.
A receiver with a curve of its own or in a
named lane gets a line under
its table too - "retries on its own curve: base 120s, cap 1800s", "is served by
relays started with --lane mail" - and the JSON carries
backoff_base_seconds, backoff_cap_seconds and lane on every receiver. So
does a declared give_up_after:
"dead-letters a deferral once its delivery has been owed for 172800s", and
give_up_after_seconds in the JSON, null where none is declared. A
scheduled receiver says when it runs - "is due 172800s after
the event occurred", or "is due at the time waitlist.events.offer_expiry
returns for the event" - and the JSON carries due_seconds for a delay and
due_callable for a callable, each null otherwise. Two fields rather than one
that is sometimes a number and sometimes a path, so a JSON reader never switches
on a value's type; a receiver with both null runs as soon as its event is
recorded.
Lines rather than columns, so the tables a project has already committed do
not change shape for properties most receivers do not have. A half of the curve
left to its setting is named as the setting rather than resolved, because the
catalogue describes declarations and the machine that builds it need not share
the relay's settings.
Building a catalogue never runs consumer code: a default_factory is named,
not called.
Who listens to what¶
from django_domain_events import listens_for, what_listens_to
what_listens_to(OrderPlaced) # [RegisteredReceiver, ...] sorted by key
listens_for("orders.reserve_stock") # RegisteredEvent | None
what_listens_to spans every mode, not only the durable ones: "who reacts to
this" is a question about the code, and an inline receiver is as much a reaction
as a queued one.
It does not repeat wildcard receivers under each event. They receive it - they
receive everything - and what_listens_to(AnyEvent) returns exactly them.
listens_for returns None for a wildcard's key, since it is owed no single
event; its delivery row already says which one it was.
listens_for is the direction an operator actually needs. A dead-letter row
names a receiver key, and the next question is always what it was supposed to be
receiving. None covers two cases that look identical from a delivery row - a
key never declared, and one whose event class was deleted out from under it.
Quiet receivers¶
"This receiver has not received anything in ninety days" is a query here, not a guess.
Driven by the registry rather than by the table, so a receiver that has never received anything appears - which is the answer worth having, and exactly the one a query over delivery rows alone cannot produce, because there is no row to find.
Only DURABLE receivers are reported: the others leave no rows, so they have no
history to be quiet about, and listing them as silent forever would train the
reader to ignore the output.
Only a succeeded delivery counts. The question is whether the receiver did its work, not whether the relay tried - a row stuck failing for a month is exactly the case this must catch.
The answer survives the prune. Deleting delivery rows would otherwise delete the evidence that a receiver ran - within minutes, for an event deleted on consumption - so the prune records each receiver's newest success among the rows it deletes, in the same transaction as the delete, and this reports the later of that record and the rows still there. A receiver that never ran is still reported as never. The record is written only by the prune, never as deliveries succeed: one row per receiver, updated by every delivery, would queue a 20,000-row fan-out on a single lock.
The window defaults to RETENTION_DAYS, the window an ordinary event's
deliveries are kept for.
The success time is read off succeeded_at, which is written on success and
never cleared - unlike completed_at, which replay and requeue clear because a
reopened row has not settled again. Reading the cleared column would tell an
operator who had just replayed yesterday's events that the receiver they were
re-running had never run at all.
Is the outbox keeping up?¶
quiet_receivers() answers whether a receiver is running; this answers
whether the queue is draining. They fail differently, and both failures are
real: a relay that has been down an hour has every receiver quiet and a backlog
climbing, while one wedged receiver has a backlog and everything else fine.
| Field | Means |
|---|---|
owed |
Deliveries due and not yet settled - pending, failed or claimed |
scheduled |
Deliveries not yet settled whose due time is still ahead - a scheduled receiver waiting. Not counted as owed |
claimed |
Currently leased by a worker |
dead |
Dead-lettered. Not counted as owed |
oldest_owed_at |
When the oldest still-owed delivery became owed: its due time where it has one, its event's recording otherwise; None when nothing is owed |
lapsed_leases |
Claimed rows whose lease has expired |
receivers |
Per-receiver backlog, worst first, omitting the ones with nothing owed, dead or scheduled. Each carries its own owed, scheduled, claimed, dead and oldest_owed_at; one with only scheduled work sorts after every receiver that is behind |
A cancelled delivery is settled, so it appears in none of these: not owed, not scheduled and not dead. An outbox holding only cancelled work reads as drained.
The age of oldest_owed_at is the number to alert on, and it is measured
from when the work arrived, not from when it next becomes available. That
distinction is the whole point: the backoff pushes a failed delivery's
available_at into the future on every attempt, so an alert written against
that column reads a negative age exactly while a receiver is failing.
Work arrives when it becomes due. For most rows that is when the event was
recorded; for a replayed or requeued row, when it was reopened; and for a
scheduled receiver, its due time. A row due in
a fortnight is scheduled until then and does not age at all, so a five-minute
backlog alert does not fire for work that is on time - and one failing after it
came due ages from its due time, not from an event a fortnight older.
It rises monotonically for as long as work sits undone - a relay that is down, a receiver that is failing its way through its retries, or a queue simply longer than the workers can drain.
It does not cover a receiver failing all the way to dead
A dead-lettered row leaves the owed set, so the backlog age drops back to
None while nothing at all is being delivered. A dead letter is
deliberately not owed - the relay will never pick it up again on its own,
and counting it would leave a backlog alert firing forever after one bad
deploy. Alert on dead as well. One number does not cover both.
Steady non-zero lapsed_leases means workers are dying mid-delivery, or a
receiver outruns its lease and has its work thrown away every time - see
lease_seconds. A relay that was asked to stop also
gives its unstarted rows back by expiring their leases, so a brief non-zero
reading after a deploy is that, not a dying worker.
The per-receiver claimed adds up to the total, and says whose rows the
workers hold. A receiver with every owed row claimed and none finishing is wedged,
or its task queue is sitting on
the messages; the same total spread across receivers is a relay at work.
Every figure here is answered from an index and none of them scans the delivery table, so the call is cheap enough to scrape on an interval.
owed means "not terminal and due", a superset of what the relay can claim at
any given moment: a row inside its backoff window is owed and not yet claimable.
The superset is the useful side - this cannot report an empty queue while work
is still outstanding. A row whose due time has not come is not outstanding yet,
which is why it is scheduled instead; at exactly its due time it is owed, as
the relay can claim it then. The prune settles by "not terminal", so it waits
for owed and scheduled rows alike.
events_status prints the same figures, scheduled among them, and a line per
receiver:
owed 3
scheduled 120
claimed 1
dead 0
lapsed leases 0
oldest owed 42s ago
billing.send_invoice owed=3 dead=0 scheduled=0 claimed=1 oldest=42s
waitlist.expire_offer owed=0 dead=0 scheduled=120 claimed=0 oldest=-
--format json carries scheduled and claimed at the top level and on each
receiver.
System checks¶
Run with python manage.py check.
| id | Level | Fires when |
|---|---|---|
E001 |
Error | A receiver listens for a class that was never @event-decorated |
E002 |
Error | The configured CODEC cannot be imported |
E005 |
Error | A declared event has a field the configured codec cannot rebuild |
E006 |
Error | WAKE is neither "notify" nor "poll" |
E007 |
Error | NOTIFY_COALESCE_SECONDS is not a non-negative number |
E008 |
Error | RELAY_PRUNE is not a bool |
E009 |
Error | RELAY_PRUNE_SECONDS is not a positive number |
E010 |
Error | PRUNE_BATCH_ROWS is not a positive whole number |
E011 |
Error | AROUND_DELIVERY is not a string, does not import, or names something not callable |
W001 |
Warning | Deliveries are owed to a receiver key the registry no longer has |
W002 |
Warning | Deliveries are owed for an event name the registry no longer has |
W006 |
Warning | A settings dict is named DOMAIN_EVENTS rather than DJANGO_DOMAIN_EVENTS |
W007 |
Warning | DJANGO_DOMAIN_EVENTS holds a key this package never reads |
E001 matters because nothing else would ever say so: the event cannot be fired,
so there is no failure to observe, only silence.
E002 runs at startup because a codec is imported lazily - without it, the first
symptom of a missing extra is a delivery failing in a worker.
E005 asks the harder half of the same question. E002 says the codec can be
imported; E005 says it can decode the events this project declares, which is
what actually fails. The asymmetry is what makes it worth a check: encoding
accepts whatever the annotation says and commits it inside the caller's
transaction, so the event exists, and only the decode refuses - in the relay,
in another process, one dead letter per receiver. A codec that does not
implement unsupported_fields is not interrogated, because this package should
not guess at what a codec it did not write can do.
W006 and W007 exist because both mistakes are otherwise silent. setting()
reads only the keys this package asks for, so a typo sits in the settings
looking effective; and the dict is DJANGO_DOMAIN_EVENTS while the app label
and most prose say "domain events", so a DOMAIN_EVENTS block returns nothing
and every value falls back to its default with nothing said.
W002 catches a renamed event, which W001 cannot see: the receivers keep
their keys, so nothing looks orphaned, while every row written under the old name
now decodes to nothing and spends one attempt budget at a time finding out. Fix
it by pinning the old identity with @event(name=...) on the class that replaced
it.
Both warnings are limited to work still owed, and both mean the same thing
by it: any status the relay would still claim - which includes claimed, since
a worker that dies leaves the row claimed under a lapsed lease. A settled row
naming a retired event is history, and warning about history on every check
run teaches the reader to skip the output.
The admin¶
Add django.contrib.admin and both models appear, read-only, with actions.
- Event records - filter by name and date, see how many deliveries are still owed per event, Replay selected events, and Cancel owed deliveries of selected events.
- Delivery records - the dead-letter queue, filterable by status and
receiver -
cancelledincluded, so withdrawn work can be listed on its own - with Requeue selected dead deliveries and Cancel selected deliveries. Each row shows its target, and the search covers the receiver key, the target and the last error, since the target is what an operator is holding when one destination of a fan-out says it never received something.
Both are read-only, and that is deliberate: the one guarantee this package sells
is that a row exists if and only if the change committed, and a form that can
write one is a way to break it. Editing status by hand is how a claimed row
gets handed to a second worker.
Deleting is refused too. It would cascade owed deliveries away with no record
that anything was lost; prune_events is the supported
route, because it re-checks settledness at the delete itself.
The requeue action goes through requeue_dead() rather than a bulk update, so it
resets the attempt budget, clears the stale lease and error, and wakes the relay.
It reports what it skipped: a mixed selection requeues only the dead rows.
Both cancel actions go through cancel_deliveries(),
so a row a worker claimed after the page loaded is left to run rather than
settled under it, and a cancel issued while a replay of the same event is open
waits for it. On the delivery changelist the selected rows are what is withdrawn
and nothing else - a sibling receiver's row of the same event stays owed - and
the message counts the rows cancelled, the ones in flight, and the ones already
settled. On the event changelist it is every owed row of each selected event,
reported as cancelled and in flight.
Every action needs the model's change permission
Django offers an action with no declared permission to anyone who can reach
the changelist, and has_change_permission gates the form alone. Replay
re-runs every durable receiver - re-sent emails, re-called webhooks, and a
fan-out receiver's targets asked for again - and a cancel withdraws work the
application is owed, so a view-only grant must carry neither. Give change_eventrecord /
change_deliveryrecord to whoever may run them; the edit form stays refused
either way.
The filters are built from the registry, not from the table: a
SELECT DISTINCT name over an event log is a full scan on every page load, and
an event declared but never fired would not appear. Selecting one and seeing an
empty list is itself the finding.