API reference¶
Shape ¶
The tables to build, and the seed that makes them reproducible.
A declaration and nothing else: it holds no connection, opens nothing, and
has no build method. Building lives in a separate function on purpose,
because a shape has to stay inert data for two things -- hashing it into a
template-database cache key, which
:func:~django_data_shape.shape_digest.shape_digest now does, and emitting
one from a real database's statistics, which is still to come. An object
that could act would be an object with state worth not hashing.
The seed is part of the declaration rather than an argument to the build, for the same reason: two builds of the same shape must produce byte-identical databases, and a seed passed at build time would let them differ while the declaration claimed they could not.
A declaration is a :class:~django_data_shape.table.Table -- so many rows,
distributed like this -- or a
:class:~django_data_shape.projection.Projection, which has no row count of
its own because its cardinality is decided by the tables it copies from. The
duplicate check below covers both kinds together on purpose: declaring one
model as a table and as a projection is the same over-determination as
declaring it twice, and it would silently mean whichever the load order
happened to reach last.
invariants are the business rules the loaded data must satisfy, checked
as SQL at the end of the build and rolling it back if any of them finds a
row. They are declared on the shape rather than on a table because a rule
worth writing down often spans two -- a child's tenant matching its
parent's, two sums agreeing across tables -- and because they all need the
same thing: every table already loaded.
The models' own constraints are read here too, which is why the
shape is where the pre-check lives rather than the table. A table knows how
many rows it declares; only a shape knows how many companies there are, and
one_active_project_per_company permits at most 50,000 is arithmetic that
needs both. See
:func:~django_data_shape.check_constraints.check_constraints.
Load order is checked here too, for the same reason. Which declaration
can be filled first is decided by the declarations alone -- a fan-out reads
its parent, a projection selects from what it names -- so a set of them that
each have to come after another is refusable without a connection. It was
the one purely structural refusal this package left until build time, which
meant a shape could be constructed, cached and passed around for a while
before saying it could never be built. See
:func:~django_data_shape.order_tables.order_tables.
invariants
property
¶
The rules the loaded data is checked against, after every table is in.
canonical ¶
The seed and every declaration, keyed by table. See Canonical.
A mapping rather than a sequence, and it loses nothing: the duplicate
check above means one table name appears once, and a dict keeps the
order it was built in. What it buys is that
:func:~django_data_shape.shape_digest.shape_digest can name the table a
refusal came from, which is the difference between "this shape cannot be
hashed" and "the compute= on orders.total cannot be hashed".
The order is kept, not sorted, unlike a table's fields. A raw
Projection names nothing it reads, so it is ordered after everything
and several of them fall back to the order they were declared in --
which means declaration order can reach the data, and a digest that
sorted it would give two different databases one key.
The invariants are in it although not one of them writes a row. A shape that reused another shape's cached template database would never run them -- the check happens during the build, and the build is what a cache hit skips -- so a rule that made no difference to the key would be a rule that silently stopped running the second time. The cost is one rebuild for a database that would have been byte-identical, which is the cheap side of that trade by a distance.
Table ¶
How many rows of one model, and how each column is distributed.
Field distributions are given as keyword arguments because that is the form
a reader scans fastest. fields= is the escape hatch, and it is not
optional politeness: a model may legitimately have a column called rows
or model, and Python's own argument binding would silently hand that
keyword to this signature instead. Without the mapping form those models
would simply be undeclarable.
statistics= maps a field name to the number of buckets the planner
should keep for that column::
Table(Order, rows=2_000_000, status=Skew(weights), statistics={"status": 500})
A target is a physical property of the column rather than of the
distribution, which is why it is declared beside the distributions and not
inside one: the same skew wants a different target in a table of two
thousand rows and a table of two million, and a caller's own distribution
would have to grow a parameter it has no use for. PostgreSQL keeps at most
target most-common values and target histogram bounds, and samples
300 times that many rows, so this is the dial that decides how much of a
declared shape the planner can actually record. A column left out keeps
whatever the schema gives it. See
:func:~django_data_shape.apply_statistics_targets.apply_statistics_targets
for what the build does with it, including the one thing it refuses.
Every refusal below happens here, at declaration time, rather than during the load. A shape that cannot describe a database should say so before it has spent a minute generating rows, and the message should name the field -- a reader who has to re-derive which column was meant has been given an error that knows more than it says.
keys
property
¶
How this table's primary keys are decided.
Never None by the time anyone can read it: _validate either inferred a strategy from the primary key's type or refused the declaration.
fields
property
¶
The declared distributions, including defaults filled in from the model.
statistics
property
¶
The per-column statistics targets this table asks the planner for.
Empty unless the declaration said otherwise, which means the columns
keep whatever target the schema gives them -- default_statistics_target
for a column no migration has touched.
columns ¶
The declared fields, in a stable order, with their model fields.
Sorted by name rather than left in declaration order: the order decides
the column list of the COPY statement, and a shape whose generated
SQL changes when two keyword arguments are swapped would hash to a
different cache key for no reason.
relations ¶
The declared relation columns, in the same stable order as columns().
computation_order ¶
The declared derivations, dependencies first.
A second order over the same columns, and deliberately not the one
columns() returns. That one is sorted by name to keep the COPY
statement stable; this one is a topological sort of what depends on
what. Computed once, when the declaration was validated, because a cycle
among derivations is a refusal and refusals belong at declaration time.
parent_fields ¶
Which of a parent's columns this table's derivations read, per relation.
The build turns this into columns on the query that already reads the parent's keys, so a child reaches its parent's values through the fan-out it already declared rather than through a lookup per row.
canonical ¶
Everything about this table that decides a row. See Canonical.
The model's label as well as its table name, because a project can rename either without touching the other and a digest that read only one would call two schemas the same.
The fields are sorted, unlike a skew's weights: they become a
COPY column list that columns() sorts anyway, so declaration
order provably does not reach a single row, and sorting means two
spellings of one declaration share a cached database instead of building
it twice.
FanOut ¶
A distribution over how many children each parent has.
Declared as a shape over an already fixed child row count, never as a multiplier. Rows times fan-out makes cardinality emergent, and a table whose size is emergent is a table nothing can assert about.
sizes supplies a relative weight per parent -- Zipf() for the
realistic heavy tail, Uniform(1, 10) for something flatter. The weights
are normalised, so their scale is irrelevant and only their spread matters.
A parent's weight is keyed on its position in the parent table, not
ordered by it. The large groups are scattered through the key range rather
than gathered at the low keys, so "the whales are the low ids" is false and
so is the reverse. That is deliberate: ordering them would put a correlation
between a parent's key and its child count into the child table, and a
correlated foreign key is planner-visible -- it would be this package
manufacturing the flattering, unreal shape it exists to remove. Reach for the
head or the tail through
:func:~django_data_shape.fan_out_sizes.fan_out_sizes, which is the
inversion the partition representation exists to make possible.
childless is the share of parents with no children at all. It is
called out separately because it is the case hand-written fixtures always
omit and the one that changes what a join does: a parent nobody references
is the difference between an inner and an outer join returning the same
thing and returning different things.
null is the share of children whose foreign key is NULL, which is only
meaningful on a nullable column. It thins the partition uniformly after
it is computed, so sizes describes the pre-null spread. Stated because
the alternative -- partitioning only the non-null children -- would make the
declared distribution mean something subtly different from what it says.
placement decides where children sit physically, and it is not
cosmetic. Emitting them parent by parent gives a perfectly clustered table
that no production system has and that flatters every index scan; the
default interleaves them the way rows really arrive.
canonical ¶
The size distribution, the parents it spreads over, and the shares. See Canonical.
parents belongs here rather than being read as a filter applied
afterwards: two shapes differing only in which parents they cover are
two different worlds, and the template-database cache keys on this. A
clone built for one tenant handed to a test asking for another would be
the cache serving a world nobody declared.
Paired ¶
A relation filled with the opposite end of an edge, never repeating a pair.
The declaration a through table needs, and the reason two
:class:~django_data_shape.fan_out.FanOut declarations are refused beside
one another. Two fan-outs partition the same rows without either seeing the
other, so nothing enumerates the pairs and whether two rows collide is a
matter of the seed. Paired is the half that looks:
.. code-block:: python
Table(
Membership,
rows=50_000,
company=FanOut(Zipf()),
person=Paired("company", Zipf()),
role=Constant("member"),
)
company partitions the rows as any fan-out does -- that is the declared
degree distribution, and it is exact. Within each of its groups, person
takes that many distinct partners, so a duplicate pair is impossible by
construction rather than removed afterwards: the same company implies a
different person, and a different company implies a different pair. The edge
count is therefore exactly rows, which is the rule this package holds to
everywhere else and which a deduplicating generator would have given up.
relation names the fan-out this one is paired with, spelled the way
:class:~django_data_shape.derivations.per_parent.PerParent names the
relation it groups by, and for the same reason: a table with three foreign
keys should not make a reader guess which one the grouping is over.
weights decides which partners are popular. It is a
:class:~django_data_shape.distributions.distribution.Distribution over the
partner table the same way a fan-out's sizes is over the parent table,
so Zipf() gives the realistic heavy tail and a flat one gives the
uniform shape this package exists to argue against.
The second side's marginal is derived, not declared, and that is on
purpose. Both marginals plus the edge count is over-determined: fixing all
three is a constraint satisfaction problem, and a CSP cannot stream into
COPY. So the edge count and one side's distribution are declared and the
other side follows -- and what follows approximates weights rather than
reproducing it. Measured against exact weighted sampling without
replacement, the busiest partner comes out about 1.1 times as busy, the 99th
percentile about 30% high, and about 7% fewer partners are touched at all.
Those numbers are in the documentation rather than only here, because a
derived shape nobody quotes is a shape nobody can check.
parents
property
¶
The partner keys this edge may use, or None for all of them.
The same narrowing
:class:~django_data_shape.fan_out.FanOut takes, and it is here for the
one thing a fan-out's version cannot do: two through tables over the
same partner model cannot otherwise be made disjoint.
Every mechanism in this package computes a column from the row index
alone, so two Paired declarations over one partner table choose
independently and overlap by construction. Any rule of the form these
two relationships must not overlap is then violated however the shape
is written -- a reviewer who is also an author, an approver who is also
the requester, an auditor auditing their own team. That family is every
separation-of-duties constraint there is, and it was inexpressible.
Narrowing the two sides to disjoint subsets is what makes it expressible, and it is also how the domains themselves work: a venue keeps a reviewer board, an organisation separates approvers from requesters. The declaration says the thing the business already says.
Keys are read through the database as a predicate rather than filtered afterwards, and a named key matching no row is refused rather than silently dropped -- both for the reasons the fan-out's version gives.
canonical ¶
The fan-out it pairs with, the weights, and any narrowing. See Canonical.
parents is part of the digest because it changes which rows exist:
two declarations differing only in their subset build different
databases, and a cache key that agreed would serve one for the other.
fan_out_sizes ¶
fan_out_sizes(
shape: Shape, model: type[Model], field: str, *, using: str = DEFAULT_DB_ALIAS
) -> ChildrenPerParent
How many children each parent got, for one declared fan-out.
The reason a fan-out is a partition of the child key range rather than a
per-child draw is that a partition can be inverted -- and until this
function existed the inversion was not reachable from outside, so a caller
who declared a skew had to GROUP BY the child table to find out where
its head was. That is an aggregate over the whole world, run inside the
session that is about to measure a query plan, to recover something the
declaration already knew.
::
counts = fan_out_sizes(shape, Order, "company")
whale, orders = counts.ranked()[0]
assert counts[whale] == orders
assert counts.childless()
It is recomputed, not remembered, and that is the design rather than an
implementation detail. The partition is a pure function of the declaration,
the seed and the parent's primary keys, so it is derived here through the
very code the build runs --
:func:~django_data_shape.resolve_fan_out.resolve_fan_out -- rather than
through a second implementation that would agree with the first only until
one of them changed.
That is also the answer to the question this function was written to
survive. A cached build skips generation entirely, so nothing carried
off a build result could be available on that path: a template database is
cloned and no row is ever generated. Recomputation does not care. The clone
holds the parent table, the declaration holds everything else, and
:func:~django_data_shape.template_database.template_database keys its
cache on this package's own version, so a template built by a release that
drew differently is never the one being read. The inversion is therefore
answerable from a cache hit, a fresh build, or a database somebody restored
from a dump, and it costs one query over the parents rather than a scan of
the children.
The one thing recomputation depends on is that the parent table still
holds the parents the children were spread across. Where the parent is
declared in the same shape -- which every cacheable shape is, since a
template is built into a freshly migrated database and a fan-out over an
empty parent is refused -- that is checked here, and a mismatch raises
:class:~django_data_shape.world_changed.WorldChanged rather than returning
a plausible partition of a world that never existed. Where the parents were
built outside the shape, by the ORM or a factory, there is nothing to check
against and nothing is claimed: the answer describes the parents that are
there now, so ask before a test starts creating more of them.
Costs one SELECT over the parent table. Everything else is arithmetic
over the parent count, which is the asymmetry worth having -- fifty
thousand parents against two million children is the shape this package is
built for, and it is the child table an aggregate would have to read.
ChildrenPerParent ¶
Bases: Mapping[object, int]
The realised fan-out: one child count per parent key.
The inversion a :class:~django_data_shape.fan_out.FanOut exists to make
possible, in the only coordinate a caller actually holds. Internally the
partition is over parent positions and child row indices, neither of
which appears in any column; this is keyed on the parent's primary key,
which is the value sitting in the child's foreign key column and the value a
test already has in its hand.
It is a plain read-only mapping, so counts[company.pk], len(counts),
sum(counts.values()) and iteration all work without anything being
invented for them. What is added is the pair of questions that are the whole
reason to declare a skew and the only ones an aggregate over the child table
could otherwise answer: which parents are the head, and which have no
children at all.
The counts are the partition. With the default null=0 that is
exactly how many child rows point at each parent. A fan-out declaring a null
share thins the partition per row after it is computed, so under one of
those the counts are an upper bound on the rows actually pointing at each
parent, and the thinning is uniform in expectation rather than exact --
:attr:null_share is the share it was thinned by, and it is zero whenever
these numbers are row counts. The ranking is unaffected either way, which is
what the head and the tail are read for.
Iteration order is the order the parent keys were read, which is the parent
table's own primary-key order. :meth:ranked is the other order, and the
two are kept separate for the reason every pair of orders in this package is:
conflating them is how "the largest group" quietly becomes "the first group".
null_share
property
¶
The share of children whose foreign key is null, from the declaration.
Zero for a fan-out that declared none, which is the case where these
counts are row counts rather than an upper bound on them. Exposed rather
than left in the declaration because a caller comparing a count here
against Child.objects.filter(parent=p).count() needs to be able to
see, from the object that gave them the number, why the two differ.
ranked ¶
Every parent and its count, most children first.
The head of the distribution is the front of this and the tail is the
back, so one method answers both rather than two answering one each with
a count argument to get wrong. A test wanting the busiest parent takes
ranked()[0]; one wanting the five busiest takes ranked()[:5].
Returned whole rather than sliced here because the interesting question is rarely only the first row: a plan assertion usually wants the busiest parent and a parent from the middle, to show that one query plan is chosen for the head and another for the body.
childless ¶
The parents with no children at all, in parent-key order.
Its own method rather than a filter over :meth:ranked because it is
the case a hand-written fixture never has and the one that changes what
a query returns: a parent nobody references is the difference between an
inner join and an outer join giving the same answer and giving different
ones. A test for that behaviour needs a parent that is genuinely
unreferenced, and guessing one from the tail of the ranking is how it
ends up testing a parent with three children.
Projection ¶
One row per pair, copied along a join, by INSERT ... SELECT.
The shape it exists for is a collection copied across an edge. A
Template has TemplateSession rows; an Event is created from a
template, and its EventSession rows mirror that template's sessions::
Shape(
Table(Template, rows=500, name=Constant("t")),
Table(TemplateSession, rows=4_000, template=FanOut(Zipf()), title=Constant("s")),
Table(Event, rows=200_000, template=FanOut(Zipf()), name=Constant("e")),
Projection(EventSession, per=Event, copying=TemplateSession),
)
That reads as one EventSession per Event, copying
TemplateSession, and it is the whole declaration: the join, the column
list and the keys are all derived from the model graph.
Why a projection rather than a vocabulary for mirroring. Three reasons, and the first is the one that decides it.
- It is what the real system already collapses into at scale. One event built from a template is a service call; a million of them is one statement. A projection is that statement, so the test database is built the way the production table would be backfilled rather than the way one row is created.
- It needs no new distribution machinery at all. A mirroring mode on
FanOutwould need an inverted fan-out, a derived cardinality and a way to say "as many as over there" -- three vocabulary items to express something that is not a distribution in the first place. - It reproduces a correlation PostgreSQL cannot see. Sessions-per-event is
correlated with the template, so every event built from a big template has
many sessions. A plain
FanOutonEventSession.eventdraws that count independently and hands the planner a join selectivity real data never has -- which is the cross-table correlation this package exists to make reproducible.
The cardinality is determined, not declared, and that is why this is not a
Table. There is no rows= here: the row count is
count(Event JOIN TemplateSession), decided by the tables already built.
Declaring it as well would be the over-determination this package refuses
everywhere else -- rows= on both sides of a one-to-one, an edge count
beside both marginals of a many-to-many, a status skew beside the fan-out
that already fixes it. The count comes back in the
:class:~django_data_shape.build_result.BuildResult like every other
table's, which is what that type was built to report.
The join is derived from the model graph. per and copying are
joined through a model they both reach in one step -- here both have a
foreign key to Template -- and per's own primary key counts as a
step of length zero, so a source that points straight at per works the
same way. Exactly one such link has to exist; zero and several are both
refused by name, because guessing which edge was meant would build a
different database from the one that was declared.
The columns are derived too, in this order, for every column of the projected model except its primary key:
- the foreign key to
pergets that row's key; - a foreign key to
copyinggets the copied row's key, which is how a projected collection records where it came from; - a column whose name matches one on
copyingis copied from it. The match is by name, so the two columns have to hold compatible types and the database says so if they do not; - a column with a plain (non-callable) Django
default=gets that value as a bound parameter, for the same reason :class:~django_data_shape.table.Tablefills one: a Django default is applied bysave()and is not DDL, so a column left out would fail its not-null check rather than quietly take it; - a nullable column, or one with a real
db_default, is left out of the statement; - anything else is refused by name, pointing at
sql=.
Where the rows land physically is decided rather than defaulted. The
statement orders by per's key and then the copied row's, which is both
deterministic -- two builds of one shape have to agree -- and the honest
physical layout. There is deliberately no placement= here, unlike on
:class:~django_data_shape.fan_out.FanOut, and the reason is a real
difference rather than an omission: a fan-out's children arrive over time,
interleaved, so grouping them is a lie. A copied collection is written in
one transaction per parent, so grouped is arrival order, and the two
orders a fan-out has to choose between are the same order here.
The keys come from the same place as every other table's. The strategy
on this declaration decides them, exactly as it does for a Table. It
just has to be able to say itself in SQL -- see
:class:~django_data_shape.keys.sql_keys.SqlKeys -- because there is no
declared row count to enumerate in Python and the rows never pass through
it. Today that means an integer key; a UUID or a caller's own function is
refused by name at declaration time rather than approximated with a
different hash, which would give one strategy two meanings depending on
which statement filled the table.
sql= is the escape hatch, and it is the whole statement's SELECT.
For anything shaped oddly -- a filter, an aggregate, a three-way join, a
window -- pass the columns and the select that fills them::
Projection(
EventSession,
columns=("id", "event", "title"),
sql=(
"SELECT row_number() OVER (ORDER BY e.id), e.id, t.title "
"FROM event e JOIN templatesession t ON t.template_id = e.template_id "
"WHERE t.title <> %s"
),
params=("hidden",),
)
The columns are field names and are checked against the model, so a typo is
refused here rather than by the database. The primary key has to be among
them: this package owns the keys, and a projection whose statement the
caller wrote has to say what they are rather than leave them to a sequence
whose current value is not part of any declaration. Nothing else about the
select is inspected -- that is what an escape hatch is -- but the build
still gives it the emptiness check, the sequence reset, the ANALYZE and
the transaction.
reads= is how such a statement says what it selects from, and it
goes with sql= rather than being an alternative to it. Nothing here
parses SQL, so a raw statement is opaque and
:func:~django_data_shape.order_tables.order_tables runs it as late as the
rest of the declaration allows. That is right until something fans out over
this table: the projection then has to run before that table, and may find
the tables it selects from still empty. Naming them puts it back in the
graph precisely -- after what it reads, before what reads it -- and it is
part of the cache key, because a statement run before and after a table
returns different rows. A derived projection has no use for it: per and
copying already are the answer.
Like :class:~django_data_shape.shape.Shape and
:class:~django_data_shape.table.Table, this is inert data: every attribute
is read-only and every derived plan is a tuple, so a declaration stays
hashable and serialisable for the template-database cache that will key on
it.
through
property
¶
The model the derived join runs on, where the caller had to say.
statistics
property
¶
The per-column statistics targets this projection asks the planner for.
A projected table needs them for exactly the reason a loaded one does. A collection copied along a join carries the source's skew into a second table, and the planner records that skew only if the column's target can hold it -- the route the rows took in has nothing to do with it.
reads
property
¶
The models this projection selects from, or nothing if it did not say.
What :func:~django_data_shape.order_tables.order_tables sorts on. A
derived projection names its two inputs, so it can be ordered after them
precisely. A statement this package did not write is opaque -- nothing
here parses SQL -- so a raw projection answers with whatever reads=
declared, and with nothing at all when it declared nothing.
max_rows
property
¶
The largest number of rows this declaration is willing to insert.
None, and no count is taken -- a declaration that does not ask is
not charged for the answer.
It exists because a projection is the one declaration with no rows=,
and that is deliberate: its cardinality comes from the join, which is
what reproduces a correlation a FanOut on the child would destroy.
The consequence is that the largest table in a database can be the one
nobody declared a size for. Its size is a product, so when both
sides of the join fan out over the same parents the busy parents
multiply: raise either declared count by four and the projection grows
by sixteen. A consumer measured 2,413,223 rows against a declaration
whose largest number was 300,000.
There is no default ceiling and there will not be one. How many rows is too many is a judgement about size, which this package does not make on a caller's behalf anywhere else either -- see the statistics target, declared for the same reason. What it can do is act on the caller's own number, before the expensive statement rather than after it.
statement ¶
The INSERT ... SELECT this declaration means, and its parameters.
Rendered here rather than stored, because quoting belongs to the connection and a declaration must not hold one. Everything the statement is made of was decided when the declaration was validated; this only spells it.
A model default lands as a bound parameter rather than as a literal in the SQL, so a string default carrying a quote is the driver's problem and not this package's.
scaled ¶
This declaration at another size, which for a projection is its ceiling.
A projection has no row count to multiply -- its size is
count(per JOIN copying), so scaling the tables it reads scales it
without anything being said. max_rows is different: it is a declared
number in the same units as that size, and a ceiling that does not move
with the factor fires on the first growth assertion.
Multiplying it by the factor is the right arithmetic rather than an
approximation of one, and for the same reason the count needs no
factor. scaled_shape scales every table, parents included, so a
parent has the same number of children at every factor; the projection
is then a sum over factor times as many parents of an unchanged
per-parent product, which is factor times the original.
Returns self when there is no ceiling, so a declaration that did not
ask for one keeps the identity it always had here.
count_statement ¶
How many rows the insert would write, without writing them.
Exact rather than estimated: it counts the same join the insert selects
from, and the select is one row per joined pair, so the two agree by
construction. The derived form counts the join directly rather than
wrapping the select -- the window function and the ordering decide what
the rows are and cost real time, while the question here is only how
many. A sql= projection is wrapped instead, because this package
cannot know what the caller's statement is one row per.
canonical ¶
Everything about this projection that decides a row. See Canonical.
The derived plan rather than only the two models it was derived from:
the join, the copied columns and the bound defaults are what the
statement actually does, and they are what changes when a model this
projection reads grows a field. A digest over per and copying
alone would call two different statements the same.
build ¶
build(
shape: Shape, using: str = DEFAULT_DB_ALIAS, *, require_statistics: bool = True
) -> BuildResult
Generate, load, reset sequences and analyze every table in shape.
The order of those steps is the whole function, and it is not
interchangeable. Loading rows into a table that was analyzed while empty
leaves the planner holding statistics from the old contents and applying
them to the new row count -- a worse lie than having no statistics at all,
and the one that produced a thirty-thousand-fold misestimate in the
measurements this package was designed from. So the ANALYZE is here, at
the end, owned by the library rather than left to the caller to remember.
A bare ANALYZE shipped in 0.1.0 because a loader that leaves its table
unanalyzed ships the exact state this package exists to condemn. What it
gathers is bounded by each column's statistics target, and that is the other
half:
:func:~django_data_shape.apply_statistics_targets.apply_statistics_targets
puts the declared targets in place first, and refuses a declaration the
planner could not record whatever it did afterwards.
require_statistics=False asks for rows and cardinality rather than for a
database the planner can reason about, and it is the only way to build on a
backend without COPY and column statistics. It is written as a
requirement being dropped rather than as work being skipped, because that is
what it does: on PostgreSQL it changes nothing at all -- the load is
still COPY and ANALYZE still runs, since both are free and leaving
them out would manufacture the unanalyzed table this package exists to
condemn. Elsewhere the rows are inserted instead and no statistics are
gathered, so cardinality is real and nothing about a plan is claimed.
The distinction is the one this package draws everywhere: generation and cardinality are backend-neutral, planner realism is not. A query count is an ORM property and means the same on any backend, which is why a growth assertion can be honest here while a plan assertion still cannot.
A :class:~django_data_shape.projection.Projection sits in the same loop
rather than in a pass of its own, and where it sits is decided by
:func:~django_data_shape.order_tables.order_tables like everything else:
after the tables it reads, and before anything that reads it. Only the step
that produces the rows differs -- one statement instead of a generated
stream -- and the three steps after it are the same steps for the same
reasons. The emptiness check, because the keys still start at 1. The
sequence reset, because the rows still sit at 1..N with the sequence at 1,
and the first objects.create() in a test would still collide. And the
ANALYZE above all, because a table filled by INSERT ... SELECT and
left unanalyzed is exactly the unanalyzed million rows this package exists
to condemn -- the route the rows took in has nothing to do with whether the
planner can see them.
BuildResult
dataclass
¶
The outcome of building a shape, table by table.
Returned rather than logged because the counts are worth asserting on: the row count a shape declares and the row count a table holds are the same number today, but they stop being the same as soon as deduplication enters the picture with many-to-many edges. Reporting achieved counts from the start means that release changes what this says, not what callers have to start checking.
TableResult
dataclass
¶
Rows loaded into one table.
Carries the database table name rather than the model, because this is what a caller prints or asserts on, and because the two stop being one-to-one as soon as through tables are generated alongside the models that declare them.
Invariants¶
PerParent ¶
Within each parent's children, one end of the group is different.
The primitive for per-group business rules, and the reason it is one primitive rather than a family: one active project per company, one default address per customer, one current period per subscription, one primary contact per account and N winners per contest are the same declaration with different words.
Table(
Project,
rows=2_000_000,
company=FanOut(Zipf(1.2), placement="grouped"),
created_at=Sequential(start, step),
status=PerParent("company", order_by="created_at", last="ACTIVE", rest="COMPLETE"),
)
The count of special rows is derived, not declared, and that is the
whole point. Fifty thousand companies and two million projects means
status='ACTIVE' matches one row per company that has any -- around 2.5%
of the table -- and that number falls out of the fan-out rather than being
chosen. Declaring Skew({"ACTIVE": 0.1, ...}) beside the same fan-out
asks for two hundred thousand active projects in a schema that permits fifty
thousand, and it is the same rule this package states everywhere else: a
distribution is declared over a fixed count, never as a multiplier. Here one
distribution is derived from another rather than declared beside it.
Assignment order is not emission order, and this is where that split
earns its keep. To say "the last project", a group has to be known; to keep
physical placement honest, rows have to be emitted interleaved. They
reconcile because a FanOut is a partition of the child range rather
than a draw per child, so a row's position within its parent's group is
arithmetic on the row index and the seed. Nothing is buffered, nothing is
sorted, and the rows still stream into COPY one at a time.
last= puts the value on the final count positions of every group and
first= on the opening ones; exactly one of the two is given. A group
smaller than count has every row special, which is arithmetically right
rather than a special case: a company with one project can have at most one
active project. A childless parent contributes nothing at all, which is
why the achieved count is one per non-empty group.
rest= is the value every other row of the group takes. It may be a plain
value, or a distribution that implements
:class:~django_data_shape.distributions.categorical.Categorical -- and
only that kind, checked here, because a distribution that could also produce
the special value would put the count back under a coin flip and undo the
derivation above. A Skew over the remaining statuses is accepted; a
Uniform is refused, because nothing can ask it what it might emit.
order_by= names the column whose ordering the group's positions agree
with. It is a claim that is checked, not a sort that is performed: see
:meth:~django_data_shape.table.Table for the two conditions that make it
true, and note that it buys realism the planner cannot see -- Postgres keeps
no statistic about which row of a group holds which value, so a shape
without it has the same selectivity and the same plan.
order_by
property
¶
The column whose ordering the group's positions claim to agree with.
canonical ¶
The relation, both ends, the count and the ordering claim. See Canonical.
order_by is in it although it changes no value in any row. What it
changes is which declarations are accepted, and a shape that would be
refused is not the same shape as one that would not -- so two of them
must not share a cached database.
Invariant ¶
A rule that must hold once the rows are in, checked after the load.
The second of the three nets, and the only one that covers rules the
database does not enforce -- which is most of them. A partial
UniqueConstraint is refused by PostgreSQL and pre-checked here; a
denormalised total, a tenant id that must match its parent's, an interval
chain with no gaps and a status history whose transitions are legal are all
rules a schema states nowhere and a generator can still break.
Two ways to say one, and they are mutually exclusive because a declaration that carried both would leave the reader to work out which one ran.
Invariant(
"a project's company matches its board's",
Project,
violated_by=~Q(company=F("board__company")),
)
Invariant(
"no company has two active projects",
sql="""
SELECT company_id, count(*) FROM testapp_project
WHERE status = 'ACTIVE' GROUP BY company_id HAVING count(*) > 1
""",
)
violated_by is a Q describing the rows that are wrong, not the
rows that are right. Stated that way round on purpose: the negation of a
rule is what a failure has to report, so writing the rule positively would
mean this package inverting a Q it did not write and reporting rows it
inferred. The queryset runs through _base_manager, for the reason a
fan-out reads through it -- a project's default manager may hide exactly the
rows an invariant exists to catch.
sql is the escape hatch, and it is a full statement rather than a
predicate because the interesting rules are aggregates: no group has more
than one, these two sums agree, this chain has no gap. Every row it
returns is a violation. A statement returning nothing passes. Nothing here
parses it, so it may read any table in the database, including ones this
shape does not build.
An invariant runs inside the build's own transaction and a violation rolls
the whole build back -- see
:class:~django_data_shape.invariant_violated.InvariantViolated for why
that is a build failure rather than a test failure.
An invariant changes no row and is still part of a shape's cache key. It has to be: the check runs during the build, so a shape that reused another shape's template database would never run it, and a rule that silently does not run is worse than no rule -- it is a rule everybody believes.
sql
property
¶
The statement whose every row is a violation, or None for the Q form.
canonical ¶
The name, the model and the rule. See Canonical.
A Q is rendered with str rather than fed in piece by piece. It
is a tree of tuples that the digest could walk, but str already
renders every child in declaration order and is what a reader would
compare two of them by -- and unlike a callable, it is a faithful
rendering: two Qs printing alike filter alike.
check_invariants ¶
Run every rule, and raise on the first that finds a row.
Called at the end of :func:~django_data_shape.build.build, inside the
transaction that loaded the rows, so a violation rolls the build back and
the database is left as it was found. Exported as well as called, because
the rules are worth running against a database this package did not build:
a template clone, a restored dump, or the state a suite has worked itself
into.
The first failure stops the run, rather than every rule being collected into one report. A generator that broke one invariant has usually broken the rules downstream of it too, and a message listing five consequences of one cause is a message that hides the cause.
Both forms run as SQL. The Q form goes through _base_manager --
never the default manager, which may filter away exactly the rows a rule
exists to catch -- and reads primary keys so the failure can quote them.
The sql form is executed as written, and every row it returns is a
violation.
Nothing is guarded here the way generation is. An invariant is supposed to query: it is the one part of this package whose whole job is a database call, which is why the refusal that governs a derivation would be exactly backwards.
check_constraints ¶
Refuse a shape whose row counts contradict the models' own constraints.
The third net -- the database itself -- catches every one of these, and
catches it with a terrible message: a unique index failing at row 700,000 of
a load that has already run for a minute, naming an index rather than a
declaration. So the arithmetic is done here instead, from
Model._meta.constraints, before a single row is generated.
_meta.total_unique_constraints is the helper that sounds right and
is the wrong one. It deliberately excludes conditional constraints -- it
exists to answer whether a relation is one-to-one, and a constraint that
only sometimes applies cannot answer that -- so it skips exactly the case
this function is for. _meta.constraints is read directly.
Three checks, and they differ in how certain they are and in what they are counting.
An unconditional UniqueConstraint is pigeonhole, and provable. Two
million rows needing distinct (company, label) pairs cannot be built
from fifty thousand companies and three labels, whatever the seed. That is
also the multi-column analysis
:meth:~django_data_shape.table.Table._check_satisfiable declines to
attempt, and it declines for a good reason: a table alone does not know how
many companies there are. A whole shape does, which is why this runs here.
Enough room is not a way to fill it, which is the second check and the one no arithmetic reaches. An unconditional constraint over two fan-outs -- the through table of a many-to-many -- passes the pigeonhole comfortably, because the product of two parent counts dwarfs the row count, and still cannot be built: two partitions of the same rows are computed independently, so the pairs they produce are an artefact of the row index and a collision is a matter of the seed. It is refused rather than counted.
A conditional UniqueConstraint is a statement about a group, and the
refusal is categorical rather than arithmetic. one_active_project_per_company
permits one ACTIVE row per company; a Skew filling status draws
each row independently, so it cannot keep a per-group rule at any weight --
2.5% is as broken as 10%, just later in the load. The arithmetic goes in the
message because it is what makes the refusal legible, not because it is what
decides it. The remedy is
:class:~django_data_shape.derivations.per_parent.PerParent, which makes
the count one per non-empty group and therefore derived from the fan-out
rather than chosen beside it.
What this cannot decide, and leaves to the other two nets. A condition
that is not a single equality -- Q(status__in=[...]), a negation, two
clauses -- is skipped, because reading it would mean this package deciding
what an arbitrary predicate matches. A constraint over expressions
rather than fields is skipped for the same reason. A conditional
constraint whose grouping columns include no declared FanOut is skipped
because there is no partition here to satisfy it with, so a refusal would
name no remedy. A column filled by a distribution that cannot enumerate
itself -- anything not
:class:~django_data_shape.distributions.categorical.Categorical -- is
undecidable, and so is a fan-out over a parent this shape does not build,
or one carrying a null share, because PostgreSQL counts each NULL as its own
group and those rows are exempt from the index. In every one of those cases
the declaration is allowed through and the post-load check and the database
are what catch it.
A :class:~django_data_shape.projection.Projection is skipped entirely. Its
columns are copied along a join rather than drawn from distributions, so
there is no declared share to compare a capacity against.
Statistics and reuse¶
apply_statistics_targets ¶
Set the declared per-column targets, and refuse a shape the planner could not record.
ANALYZE has shipped since 0.1.0 and it is only half the story. What it
gathers is bounded by each column's statistics target: PostgreSQL keeps at
most that many most-common values and that many histogram bounds, and samples
300 times as many rows to find them. Everything past the target is collapsed
into one residual frequency, so a column with more distinct values than its
target has a shape the planner cannot see -- and this package exists to make
a declared shape visible.
The target is declared, never inferred. This function reads the
distributions only to refuse, and that distinction is the whole design.
Raising a target on the caller's behalf would be this package choosing how
the planner sees a column, silently, on evidence the declaration does not
contain: a hundred-value skew wants a hundred buckets in a table where those
hundred values are the query's predicate and wants far fewer where they are
not, and nothing in a distribution says which. So a shape that gets a
hundred-bucket histogram because default_statistics_target happens to be
a hundred is treated as different from one that asked for it -- not by
guessing at the second, but by making the first impossible to hold by
accident. Either the declaration can be recorded, or the build stops and
names the column.
Which is a refusal that cannot happen at declaration time, like
:class:~django_data_shape.derivations.given.Given's missing case and a
:class:~django_data_shape.projection.Projection that inserts nothing. The
number it compares against lives in the server -- a column carries a target
set by a migration, and everything else falls back to a setting an operator
can change -- so the declaration alone cannot know it. It is still raised
before a single row is generated rather than after the load, because a
refusal that costs a two-million-row COPY first is a refusal nobody
thanks you for.
The order matters and is easy to get backwards. A target changed after
ANALYZE has run does nothing at all until the next one, exactly as
ANALYZE before a load leaves stale statistics behind. So this runs before
the rows, and the ANALYZE at the end of the build is the one that reads
it -- the ordering is owned by the library for the same reason the rest of
the sequence is.
Nothing happens on another backend. The branch is on the connection's vendor rather than on a failed statement, so it is covered by passing a vendor rather than by running the suite on the backend it skips.
shape_digest ¶
A hexadecimal digest of everything in shape that decides the data.
The cache key the whole template-database mechanism rests on: two shapes
that would build the same rows hash the same, two that would not do not, and
the answer is the same in every process. That last clause is the one worth
saying out loud, because Python's own hash() does not satisfy it -- it is
salted per interpreter run for strings and bytes, so a key built on it would
miss the cache every time in the best case and, if the salt ever agreed
across two runs of different shapes, serve the wrong database in the worst.
This uses BLAKE2b for the same reason
:func:~django_data_shape.utils.field_stream does.
Everything reachable from the declaration contributes: the seed, each table's model and row count, every distribution and its parameters, every fan-out with its childless and null shares and its placement, every derivation, every projection's derived statement, the key strategies and the statistics targets. A declaration that changes any of them is a different database and gets a different digest.
Two things are hashed in declaration order rather than sorted, because
the order is part of what the declaration means. A
:class:~django_data_shape.distributions.skew.Skew lays its cumulative
bounds out in the order its weights were given, so the same weights in a
different order assign different values to the same draw. And a shape's
tables keep their order, because a raw
:class:~django_data_shape.projection.Projection is ordered after
everything and ties among several fall back to the order they were declared
in. Sorting either would let two different databases share a key. Where an
order provably does not reach the data -- a table's fields, which are sorted
into a COPY column list before anything is generated -- the sorted form
is hashed instead, so two spellings of one declaration share a key.
What it refuses, and the reason it refuses rather than approximates.
:class:~django_data_shape.derivations.derived.Derived and
:class:~django_data_shape.keys.key_function.KeyFunction hold a callable the
caller supplied. There is no honest digest of a callable: two lambdas share a
name, a closure carries values from somewhere else, and a function hashed
down to its bytecode still changes what it returns when a module-level
constant it reads is edited. Every one of those failures is in the same
direction -- the digest agrees while the data has changed -- and the result
is a suite silently running against a database built from code that no longer
exists. So a declaration this cannot read raises
:class:~django_data_shape.unhashable_shape.UnhashableShape, naming where it
is. The same refusal covers a value type it does not understand, for the same
reason: a value it cannot encode is a value it would have to leave out.
The way out is not a flag. A consumer whose own declaration really is data
implements :class:~django_data_shape.canonical.Canonical and joins in; one
that wraps a callable builds with :func:~django_data_shape.build.build
directly and pays the load, which is the honest price of a shape whose data
this package cannot recognise twice.
Canonical ¶
Bases: Protocol
A declaration that can describe itself as plain, comparable data.
The third opt-in protocol in this package, and written for the same reason
as the other two. :class:~django_data_shape.distributions.bounded.Bounded
lets a distribution say how many values it can produce;
:class:~django_data_shape.keys.sql_keys.SqlKeys lets a key strategy say
itself in SQL. This one lets any declaration say what it is made of, so a
whole :class:~django_data_shape.shape.Shape can be hashed into a
template-database cache key -- see
:func:~django_data_shape.shape_digest.shape_digest.
canonical returns a tree of plain values: numbers, strings, None,
Decimal, dates and times, UUIDs, enum members, sequences, mappings,
and other objects implementing this protocol. Anything else is refused when
the digest is taken, by name, because a value the digest cannot read is a
value it would have to ignore -- and an ignored value is one that changes
the data while the cache key stays the same.
Order is preserved rather than sorted, and that is deliberate. A
:class:~django_data_shape.distributions.skew.Skew walks its weights in
declaration order to place the cumulative bounds, so two skews with the same
weights in different orders assign different values to the same draw. A
digest that sorted them would call two different databases one, which is the
one direction this must never be wrong in. Where an order genuinely does not
reach the data -- a table's field mapping, which is sorted before it becomes
a COPY column list -- the implementation sorts, and says so.
What deliberately does not implement it.
:class:~django_data_shape.derivations.derived.Derived and
:class:~django_data_shape.keys.key_function.KeyFunction each wrap a
callable the caller supplied, and a callable is code rather than data. Its
name is not its behaviour: two lambdas are both <lambda>, and even a
function hashed byte for byte can change what it returns when a constant it
reads is edited somewhere else. A digest that agreed while the data changed
would serve a stale database, so those two are refused by name instead --
the same choice SqlKeys makes for a projected table's keys, and for the
same reason: approximating gives one declaration two meanings.
A consumer whose own distribution, derivation or key strategy is pure data implements this and joins in; one that wraps a callable should not, and the refusal will name it.
template_database ¶
Make sure a database holding shape exists, and return its name.
The expensive half of the cache, and it runs once per machine rather than
once per test run. Measured on the two-million-row table this package was
designed against: generating and COPY-loading it is about nineteen
seconds, and :func:~django_data_shape.clone_database.clone_database turns
the result into another database in 174 ms. Everything below exists to make
the first number payable once and the second one the one a suite pays.
The name is a digest of everything that decides the contents, which is what makes reuse safe rather than merely fast:
- the declaration, through
:func:
~django_data_shape.shape_digest.shape_digest; - the schema it is loaded into -- every migration on disk, and every installed model's table, columns, types and nullability, so that a project whose apps have no migrations is covered too;
USE_TZandTIME_ZONE, because every value goes through its field'sget_db_prep_saveand a datetime column lands somewhere else under a different one;- this package's own version, because a release that changes how a distribution draws changes the rows without changing the declaration.
Change any of them and the name changes, so the old database is simply not
asked for again. One thing that is not covered, stated rather than
left to be discovered: editing a RunSQL inside a migration that already
exists changes the schema while leaving the migration's name and every
model's fields alone. Drop the template by hand --
:func:~django_data_shape.drop_database.drop_database -- when that happens.
What it does not support, and why:
- Anything but PostgreSQL.
CREATE DATABASE ... TEMPLATEhas no equivalent elsewhere, and the answer to "is a table set or a database the unit of reuse" is a database precisely because that statement exists. - A shape whose declaration cannot be hashed. A
:class:
~django_data_shape.derivations.derived.Derivedor a :class:~django_data_shape.keys.key_function.KeyFunctionwraps a callable, and hashing code as though it were data is how a cache serves a database built from a function that has since been edited. Those shapes raise :class:~django_data_shape.unhashable_shape.UnhashableShapeand are built with :func:~django_data_shape.build.buildinstead. - Being called inside a transaction. Filling the template means pointing the connection at another database and closing it, and closing a connection inside an atomic block leaves it unusable for the rest of the block. This belongs in session setup, before any test has opened one, and says so rather than poisoning the connection.
- A template on a different server from the database that will clone it.
CREATE DATABASE ... TEMPLATEcopies files on one cluster; there is no cross-server form, and nothing here pretends otherwise. - Cleaning up after itself. A template is a cache on a machine, keyed by content, so nothing that survives is ever wrong -- only unused. Deleting on a guess would mean dropping a database because this package no longer recognised its name.
Parallel runs are supported, and that is what the advisory lock is for.
Under pytest-xdist every worker asks for the same template at the same
moment; without a lock each would find it missing and each would build it.
The lock is taken on the digest, so workers wanting different templates never
wait on each other, and it is held on the maintenance connection so it is
released even if the process dies. Cloning is not serialised by anything
here -- PostgreSQL handles concurrent copies of one source itself.
Connections to a finished template are turned off with ALLOW_CONNECTIONS
false, because the single failure mode of the whole mechanism is
PostgreSQL refusing to copy a database somebody is attached to. Turning them
off is the difference between that being impossible and it being unlikely.
To look inside one: ALTER DATABASE <name> ALLOW_CONNECTIONS true.
clone_database ¶
clone_database(
template: str,
target: str,
*,
using: str = DEFAULT_DB_ALIAS,
strategy: str | None = "file_copy",
replace: bool = False,
) -> None
CREATE DATABASE target TEMPLATE template, which is the cheap half of this package.
The operation the whole template cache exists to reach. Building a shaped
database is expensive and copying one is not: measured on a 212 MB database,
the build was about nineteen seconds and this is 174 ms with
STRATEGY = file_copy, against 0.85 to 1.45 s on PostgreSQL's default
wal_log. The statistics come with it -- pg_statistic rows and the
per-column targets in pg_attribute are ordinary catalogue contents, so
the clone is planner-ready without a second ANALYZE.
strategy defaults to file_copy because that is the whole point.
wal_log writes every copied page through the write-ahead log, which is
what makes a clone crash-safe and point-in-time recoverable -- properties a
test database created fresh each session has no use for, paid for at five to
eight times the cost. None leaves the clause off and takes the server's
own default, which is the setting for a PostgreSQL older than 15.
Nothing may be connected to the template while this runs. PostgreSQL
refuses to copy a database that has other backends attached, with "source
database is being accessed by other users", and the usual cause is the
process doing the cloning: a Django connection left open from building it,
or a psql somebody forgot. That is why
:func:~django_data_shape.template_database.template_database closes its
connection and turns connections off on the finished template rather than
trusting nobody will open one. Concurrent clones of one template are fine
and are what a parallel test run does -- PostgreSQL serialises them.
replace=True drops target first. It is not the default because this
function destroys a database, and a default that destroys is a default
somebody meets by accident; a test-database setup that reruns wants it, and
says so.
Both names are quoted by the connection rather than interpolated raw, and
the strategy is chosen from a fixed set, because none of the three can be a
bound parameter: CREATE DATABASE is a utility statement whose grammar
has no placeholders.
drop_database ¶
Drop name if it is there, and say whether it was.
Public for two reasons, and the second is the one that makes it worth a
module. A test-database setup that clones per session has to remove the
clone again, and writing that as raw SQL in a conftest.py means writing
the autocommit rule and the quoting by hand.
And a template database is never removed automatically. It is a cache on
a machine, keyed by content, so a template that no shape asks for any more is
a template nothing will ever open again -- and deleting it on a guess would
mean this package dropping a database because it did not recognise a name.
So they accumulate, and this is how a stale one goes. They are named
data_shape_ followed by a digest; psql -c "\l data_shape_*" lists
them.
ALLOW_CONNECTIONS being off on a finished template does not get in the
way: DROP DATABASE does not connect to what it drops. What does get in
the way is somebody else's open session, which PostgreSQL reports by name.
require_clone_strategy ¶
Refuse a strategy this server cannot take, and one it has never heard of.
Its own function for the reason
:func:~django_data_shape.require_postgres.require_postgres is: it reads
pg_version off the connection and nothing else, so the refusal is
covered by passing a version rather than by installing a PostgreSQL 14 to
run the suite against. A degradation path reachable only on the
configuration it degrades for is a path this package's coverage gate cannot
see, and that gate is on PostgreSQL precisely because that is where the work
happens.
None is always allowed: it means no clause at all, which is what every
version does when nobody asks for a strategy.
The pytest surface¶
shape_fixture ¶
A session-scoped pytest fixture that builds shape once.
Bind it to a name in conftest.py and request that name from a test::
from django_data_shape import Constant, Shape, Table
from django_data_shape.fixtures import shape_fixture
orders = shape_fixture(Shape(Table(Order, rows=100_000, status=Constant("new"))))
::
import pytest
@pytest.mark.django_db
def test_the_dashboard_query(orders):
assert orders.rows == 100_000
It composes with pytest-django rather than replacing it. The fixture
requests django_db_setup, which is the seam a project overrides to
decide how its test database is made, so whatever a project has done there
-- a template database, --reuse-db, a different creation strategy -- has
already happened before a row is generated. It then writes through
django_db_blocker.unblock(), the mechanism pytest-django documents for
populating a database once. Neither of those is imported: they are asked for
by name, so this package depends on two fixture names and not on
pytest-django's internals.
Session scope is load-bearing, not a performance choice. pytest creates
higher-scoped fixtures before lower-scoped ones, so a session-scoped build
always runs before the function-scoped db fixture opens the transaction
that wraps a test -- which is what makes the rows committed and visible to
every later test. A function-scoped build would be ordered against db by
the accident of argument order, and on the losing side of that order it
would be rolled back with the test that happened to build it.
Yields the :class:~django_data_shape.build_result.BuildResult, so a test
can assert on the size of the world it was handed.
One caveat, and it is worth stating plainly: a test marked
django_db(transaction=True) truncates every table at teardown, and takes
the session's rows with it. Nothing rebuilds them, so a later test that
reads this fixture is measuring an empty database. Keep transactional tests
off the tables a shape owns, mark them serialized_rollback=True, or
build per test with
:func:~django_data_shape.scaled_world.scaled_world at factor 1 -- which
undoes itself and therefore does not care.
One world per table. A session world holds its rows for the whole run,
so :func:~django_data_shape.fixtures.scale_fixture.scale_fixture over the
same model cannot build: the second build meets a table that is not empty
and is refused. Give the two different models -- the session world the tables
a plan assertion needs to be big, the scale harness the tables a growth
assertion counts. It is the first thing a consumer composing both hits, and
the refusal now names it.
On a connection that cannot carry a shaped database the fixture skips with the reason rather than raising, so a suite that also runs on SQLite reports what it did not check instead of erroring or, worse, passing.
scale_fixture ¶
A pytest fixture yielding a :class:~django_data_shape.scale_protocol.ScaleProtocol.
The pytest face of the scale protocol. Bind it in conftest.py::
from django_data_shape import Constant, Shape, Table
from django_data_shape.fixtures import scale_fixture
world = scale_fixture(Shape(Table(Order, rows=100, status=Constant("new"))))
and a growth assertion has somewhere to ask for a bigger world::
def test_the_dashboard_query_is_constant(world, django_assert_num_queries):
for factor in (1, 10):
with world(factor):
with django_assert_num_queries(3):
dashboard()
The declared row counts are the world at factor 1, so the base declaration should be the smallest world that still means something -- a hundred rows against a thousand is the regime this is for, and it is milliseconds per factor. Size, in the two-million-row sense that makes a query plan realistic, is a different assertion with a different cost and does not vary a factor at all.
Function-scoped, and it requests pytest-django's db fixture, which does
two things worth knowing. A test using this needs no django_db marker of
its own. And each world is then built inside the transaction that wraps the
test, so tearing it down is a savepoint rollback: cheap, exact, and leaving
the test's own transaction usable afterwards. A test that marks itself
transaction=True still works -- the marker wins over the fixture, and
the rollback is then an ordinary one.
Not over a model a session world already holds. Each world here is built
from empty and undone again, so a table that
:func:~django_data_shape.fixtures.shape_fixture.shape_fixture filled for
the session is one this cannot build into at all -- the rows are still there,
and the build is refused. The two compose over a graph by taking different
models, not by taking turns over one.
Open a query capture inside the block, never around it. Repeated here
from :func:~django_data_shape.scaled_world.scaled_world and not merely
cross-referenced, because the person who can make this mistake is the one
writing the test, and they reach this function without ever opening that
one. Building a world emits statements of its own::
with world(factor):
with django_assert_num_queries(3): # inside
dashboard()
A capture wrapped around world(factor) counts the build as well as the
block. On PostgreSQL that is a fixed overhead -- sixteen statements for a
two-table shape, at every factor. Off PostgreSQL the rows go in as ordinary
inserts, one statement per thousand, so the count grows with the factor
and the assertion reads the loader's growth curve instead of its subject's.
It works on any backend Django supports, because what a growth
assertion measures -- the number of queries a block emits -- is an ORM
property rather than a planner one. Where the backend has COPY and
column statistics the world is built with them; where it does not, the rows
are inserted and no statistics are gathered, so the cardinality is real and
nothing about a plan is claimed. That is the one place this package builds
outside PostgreSQL, and it is allowed precisely because the assertion it
serves does not need the planner. A plan assertion still skips: see
:func:~django_data_shape.fixtures.skip_unless_postgres.skip_unless_postgres.
skip_unless_postgres ¶
Skip the current test, with a stated reason, where operation cannot mean anything.
The pytest twin of
:func:~django_data_shape.require_postgres.require_postgres, and the two
differ only in how the same sentence is delivered. A function that is called
raises; a fixture that cannot supply what it promised skips, because a test
that never ran is honest and a test that ran against a database nobody
shaped is not. Silently yielding an unbuilt world would be the vacuous pass
this package exists to expose, and returning a warning would be one nobody
reads.
The reason is the refusal's own message rather than a shorter one written here, so the skip line names the same three things the exception does: what was refused, which connection, and what that connection actually is.
Public because a consumer writing its own fixture over a shaped database needs exactly this, and because a plan assertion in a downstream package has the same obligation to skip rather than pass.
ScaleProtocol ¶
Bases: Protocol
Make the world be at factor, then let the caller run its block.
The seam between this package and a consumer that asserts a query count is
O(1) rather than O(N) by running one block at several scale factors.
Such a consumer depends on this shape, not on this package: anything
callable as at(factor) returning a context manager will do, so a project
on a backend this package refuses -- or one that has not adopted it yet --
supplies its own five-line callable and the assertion works unchanged.
:func:~django_data_shape.fixtures.scale_fixture.scale_fixture yields an
implementation of it, and
:func:~django_data_shape.scaled_world.scaled_world bound to a shape is
another.
Two details are deliberate.
The factor is positional-only, and that is not a detail. A structural
type matches parameter names as well as types unless a parameter is marked
positional-only, so without the / the protocol would have accepted only
implementations that happened to spell the argument factor -- a protocol
about this package's naming rather than about the shape of the call, and one
that rejected the five-line callable the paragraph above offers. Callers pass
the factor positionally; implementations name it whatever reads best.
It is a context manager, not a plain call. A world has to be taken down as well as put up, and only whatever built it knows how to undo it. A protocol that only built would leave every implementation to invent its own teardown, and the one this package uses -- roll back to the savepoint the world was built inside -- is not something a caller could arrange from outside.
What it yields is a row count or nothing, never one of this package's
types. The value is how many rows the world actually holds, and it is a
diagnostic rather than the growth curve's x-axis: the caller passed the
factor in and already knows it. Yielding a BuildResult would have been
richer and would have made the protocol unimplementable by anyone who has
not installed this package, which is the opposite of what a seam is for.
None is allowed for the same reason the value is optional in spirit
already. The five-line callable the paragraph above offers is the one a
consumer writes first::
@contextmanager
def world(n: int) -> Iterator[None]:
build_my_fixtures(100 * n)
yield
and it has no count to report, only rows. Requiring one would have made the
invitation false -- which it was, in this exact way, until the type below
was widened. A caller reading the value has to tolerate None; an
implementation that can count cheaply should still yield the number, because
a growth curve annotated with what the world actually held is worth more
than one annotated with what was asked for.
Worth recording, because this is the second time the docstring promised
more than the signature allowed: the first was the parameter name, fixed by
making factor positional-only, and both were found by a consumer rather
than by review. A type-level promise with no type-level test behind it is
what let each of them ship, so
tests/scale_protocol_consumers.py now carries the invited implementation
itself and the suite type-checks it.
Spelled without this class -- for a consumer who would rather restate the shape than import it -- it is exactly::
Callable[[int], AbstractContextManager[int | None]]
Given here so that restatements converge on one, rather than on a looser
ContextManager[Any] that would accept things this does not.
scaled_world ¶
Build shape at factor, run the caller's block, then undo it.
The implementation of :class:~django_data_shape.scale_protocol.ScaleProtocol
for a project that uses this package. Bound to a shape it is one::
world = functools.partial(scaled_world, Shape(Table(Order, rows=100)))
for factor in (1, 10):
with world(factor) as rows:
...
Yields the number of rows the world holds, which is what the database took rather than what the declaration asked for -- the two are the same today and stop being so once deduplicated many-to-many edges arrive, and a growth curve annotated with a number nothing achieved would be worse than one annotated with none.
It does not require planner statistics, and that is what makes it
portable. A growth assertion counts queries, and a query count is an ORM
property that means the same on any backend -- so this builds wherever there
are rows to build, using COPY and ANALYZE where the backend has them
and plain inserts where it does not. Nothing about a plan is claimed on a
backend that cannot support the claim, which is the same line this package
draws everywhere: generation and cardinality are backend-neutral, planner
realism is not. A plan assertion still belongs behind
:func:~django_data_shape.fixtures.skip_unless_postgres.skip_unless_postgres.
Open a query capture inside the block, never around it. Building a world
emits statements of its own, and a capture wrapped around world(factor)
counts them along with the block's. On PostgreSQL that is mild and fixed:
sixteen statements for a two-table shape at every factor, because COPY
does not go through Django's execute_wrapper and only the emptiness
check, the statistics-target read, the parent key read, the sequence reset,
the ANALYZE and the savepoints do.
That fourteen is counted with CaptureQueriesContext -- what
django_assert_num_queries reads -- inside a non-transactional django_db
test. Both halves of that sentence move the number: the same shape counted
through execute_wrapper, which is what a capture built on that hook sees,
is eleven, because the savepoints and the emptiness check reach the query log
by a route the wrapper does not; and a transaction=True test drops one
more savepoint. So do not read the absolute figure as a constant of this package.
What is invariant, and what the tests below pin, is the shape of each: fixed
on PostgreSQL whatever the factor, growing off it. Off PostgreSQL it is neither: the inserts are ordinary
statements, one per thousand rows, so the captured count grows with the
factor -- and a growth assertion measuring from outside the block would
read the loader's own curve as its subject's.
Both halves of that are pinned by tests rather than left as prose, because a measurement in a docstring is the first thing to rot and the consumer this matters to cannot check it without taking the dependency the protocol exists to avoid. The number above was already wrong once, for exactly that reason.
The teardown is a rollback, not a delete. Building inside a transaction
and rolling it back at the end restores exactly the state the block started
from, which matters twice: this package never issues a destructive statement
against a table it did not fill, and inside a pytest-django db test the
rollback is to a savepoint, so it costs nothing and leaves the enclosing
test transaction usable afterwards. Outside one it is an ordinary
transaction rollback, so the same code is correct in both places.
One thing the rollback does not undo, because the database will not: an
identity sequence moved past the keys a build assigned stays moved, since
setval is not transactional. Nothing here reads it -- keys are assigned,
not drawn -- so the only visible effect is that a row created by the ORM
after a world is torn down gets a larger id than it otherwise would.
scaled_shape ¶
shape with every declared row count multiplied by factor.
This is the answer to the question a growth assertion asks -- make the world be at factor F -- and the choice worth stating is that a factor varies the declaration. The alternative, and the one that looks cheaper, is to build once at the largest factor and let a smaller factor see only part of it. Three things say otherwise:
- A subset is not a smaller database; it is the same database with a filter. The table still holds every row, the statistics still describe every row, and an index still spans every row. Worse, the block under test would have to cooperate -- to restrict itself to the subset -- so the harness would leak into the code being measured. A growth assertion whose subject has to know it is being scaled is measuring the harness.
- A fan-out is a partition of the child key range, so taking a subset
changes the shape rather than the size. Cutting the children short
removes whole parents under
groupedplacement and thins every parent underarrival, so the childless share and the tail -- the two things the declaration exists to state -- would come out different at every factor, and the curve would be fitted over worlds of different shapes. Multiplying the row counts keeps the distribution and varies only its size, which is what "the same world, bigger" has to mean. - A shape is inert, hashable data, and a scaled shape is another one. That is the representation the template-database cache will key on, so each factor gets a cache key for free and caching makes a repeated factor cheap without changing this protocol. A subset has no key of its own; it is a query over somebody else's build.
The cost objection does not survive the numbers either. Growth assertions run at small absolute scales -- a hundred rows against a thousand -- where building is milliseconds and where the question is the shape of the count curve rather than plan realism. The two-million-row build is the plan assertion's problem, and plan assertions do not vary a factor.
Every table is scaled, parents included, and that is the point rather than a simplification: scaling only the child table would change the average fan-out along with the size, so the two worlds would differ in a second way and the curve would no longer be about size alone.
One factor moves every dimension at once, and there is deliberately no
per-table factor. A consumer's most useful finding is often "this count is
O(parents), not O(orders)", which a single factor cannot name because both
axes move together. The answer is not a scale=False on Table: holding
the child count fixed while the parents grow also changes the average
fan-out, so a curve measured that way mixes two causes, and a boolean would
hide exactly the confound the reader needs to see. The honest way to vary one
dimension is to write the declaration that varies it -- and nothing has to
change for that, because the protocol takes a callable rather than a
shape. "Which dimension varies" is a property of the function a consumer
binds, so the seam already supports it and would not need to be reopened.
Recording that as a decision rather than an oversight: a per-table factor is purely additive, so it can be added the day a real curve needs one, while freezing a meaning for it now would answer a question -- what a pinned parent means for a fan-out declared over it -- that no measured case has asked yet.
The scaled tables are rebuilt through Table's own constructor rather
than assembled behind it, so a declaration that is only valid at its
original size is refused at the factor that breaks it -- naming the factor,
because a message about a row count the caller never wrote is a message that
knows more than it says.
A :class:~django_data_shape.projection.Projection needs no factor for
its size, and one for its ceiling. It has no declared row count to
multiply: its size is count(per JOIN copying), so scaling the tables it
reads scales it by exactly the same amount without anything being said.
That is the determined-not-distributed property paying for itself -- a
mirroring vocabulary with a row count in it would have needed a rule here,
and would have had to pick between scaling the count and scaling the thing
the count was derived from.
max_rows is multiplied, and the same reasoning is why. A ceiling is
a declared number in the same units as that size, so one that stayed put
would fire on the first growth assertion -- which is exactly what a consumer
hit within a run of asking for the feature. Because every table scales,
parents included, a parent has the same number of children at every factor
and the projection is a sum over factor times as many parents of an
unchanged per-parent product: linear, so multiplying the ceiling is the
arithmetic rather than an approximation of it.
shape_from_factory ¶
shape_from_factory(
factory: Callable[..., Model],
*,
samples: int = 200,
defaults: Mapping[str, object] | None = None,
using: str = DEFAULT_DB_ALIAS,
) -> str
Run a factory, measure what it made, and return a declaration as source.
Source to read, edit and check in -- never a Shape to build from, and
that is the whole design rather than a limitation of it. A shape this package
builds is declared, which is what makes it reviewable and assertable; a
shape learned from a sample is neither, and one used directly would change
whenever the factory did, silently. So what comes back is text, and a person
decides what of it to keep.
Because the thing it usually finds is that the factory is flat. Factories
are written for single-object tests, so they fix values: one status, one
parent, one count. Measured on exactly that shape, faithful inference emits
status=Constant('ACTIVE'), company=Constant(1) -- the uniform world this
package exists to argue against, now with a declaration blessing it. So every
such column is reported as a finding rather than quietly written down,
and the report leads with them.
A sub-factory is the sharpest case and the one worth running this for.
company = SubFactory(CompanyFactory) creates one parent per child, which
is a fan-out of degree one: every parent has exactly one row, the average is
the truth, and the join estimate cannot miss. It is the single most
unrealistic thing a fixture can do and it is invisible in the factory's own
source, so it is detected by watching which other tables grew and by how
much.
defaults is passed to every call, because most factories in a mature
codebase need arguments. A TeamFactory wanting a permission_role,
called with nothing, leaves a required column empty and fails as a raw
IntegrityError naming a constraint -- which says nothing about what the
caller did. Anything the factory raises is caught and named here instead,
with the call number, because "it failed" and "it failed after three" are
different bugs.
Nothing is left behind: the calls run inside a transaction that is rolled back, so this can be pointed at a development database without writing to it.
samples decides how much of the tail is seen, and the answer moves with
it -- fifty runs saw nineteen distinct parents where a thousand saw
forty-nine. That is stated in the output rather than hidden, because two
people running this at different sizes should not be surprised by two
different declarations.
Derivations¶
Derivation ¶
Bases: Protocol
Computes one column from values that are already known.
Deliberately not a kind of
:class:~django_data_shape.distributions.distribution.Distribution, and the
difference is not a technicality. A distribution answers what is the
marginal shape of this column across N rows -- which is the question the
query planner asks, and the only kind of answer that decides a plan. A
derivation answers given this row's other values, what is this one, which
is what a creation service encodes and what no planner can see. Keeping them
separate types is what keeps the first kind enumerable later, when a shape
has to be summarised for a statistics target or a cache key: a mechanism
that let a derivation masquerade as a distribution would make that
enumeration quietly wrong.
Three members, and only the first is what varies between the faces:
scope
Where sources are read from. See :class:Scope.
sources
The names to resolve, in the order value will receive them.
Resolution belongs entirely to the caller, so an implementation never
touches a plan, a connection or another column.
value
The value itself, from the row index, this column's own draw, and the
resolved sources. draw is supplied even to implementations that
ignore it, for the same reason
:class:~django_data_shape.distributions.distribution.Distribution
supplies both halves: it keeps the protocol single-shaped, so the
resolver has one call to make and not three.
Like a distribution, an implementation must be a pure function of its arguments. One that carried state between calls would make the same shape produce different data depending on the order rows were computed in, and computation order is precisely what this mechanism reserves the right to choose.
Scope ¶
Bases: Enum
The one thing that distinguishes one derivation from another.
A derivation computes a column from something already known. What varies between the useful kinds is not how the computation runs but where its inputs are read from, and that is the whole of this enum. It is the reason there is one mechanism here rather than four: correlate-with-the-parent, correlate-with-a-rank and compute-from-this-row differ by this value alone, and by nothing in the resolver.
ROW
Sources are other declared columns of the same row, named plainly:
"quantity". They are computed first, which is why a derivation
needs a computation order of its own -- the column order exists to keep
the COPY statement stable and says nothing about dependencies.
PARENT
Sources are columns of the row on the other side of a declared
FanOut, named "relation.field": "account.signed_up_at".
They are read out of the parent table, not recomputed from the
parent's declaration, so a parent built with the ORM works exactly like
one built here. That is the same correction the fan-out itself took:
the keys are queried rather than assumed, and so are the values beside
them.
RANK
Sources are the names of shared ranks, invented by the declaration:
"size". A rank resolves to a draw in [0, 1) that is the same for
every column naming it, which is what makes two columns extreme in the
same rows. Ranks are per table and per row, because a rank shared across
tables would be aligning entities that have nothing to do with each
other.
GROUP
Sources name a declared FanOut, and each resolves to a pair: this
row's position inside its parent's group of children, and how many
children that group has. It is what makes a per-group business rule --
one active project per company -- satisfiable while rows are still
emitted interleaved, because the fan-out is a partition of the child
range, so both numbers are arithmetic on the row index rather than
anything that needs a group held in memory.
The pair is a plain ``(position, size)`` tuple rather than a type of its
own, because it is a resolved source rather than a declaration: nothing
outside the resolver and the derivation reading it ever sees one, and a
class here would be a public name for an argument.
Derived ¶
compute over the named sources, in the named scope.
The general case, and the one every other face is a shorthand for. A consumer who wants "call my own code to fill this column" wants exactly this, which is why it is the mechanism rather than a fourth thing beside three correlation primitives: built separately, custom logic and correlation become two vocabularies that overlap on the interesting half.
scope is the parameter, not a family of classes. The default reads other
columns of the same row; Scope.PARENT reaches the row across a declared
FanOut and Scope.RANK reads a shared rank. So a consumer's own
function can correlate with the parent without this package shipping a face
for their particular correlation:
Your function may not touch the database. That is not a request: the
generation pass runs under a wrapper on the connection being built, and a
query raises
:class:~django_data_shape.derivation_queried_database.DerivationQueriedDatabase
naming the table. The rule is what keeps this a derivation rather than the
per-row creation hook this package exists to replace -- a hook whose body
can query is a hook whose body will, and then nothing is COPY-loaded and
this is a slow fixtures library with extra vocabulary.
compute receives the resolved sources positionally, in the order they
were declared, and nothing else. Not the row index, and not a draw: a
function of the row index is a
:class:~django_data_shape.distributions.sequential.Sequential and a
function of a draw is a distribution, and both of those are already
planner-visible declarations. Handing a derivation the same inputs would
make it possible to write a distribution that the planner-facing half of
this package cannot see.
After ¶
parent.field plus a gap of at_least up to at_least + within.
An order is created after its customer signed up; a payment settles after its invoice was issued. Left undeclared, the two dates are independent, and a date-range join over them has a selectivity no production database has -- every combination occurs, including the half that cannot.
Table(
Order,
rows=2_000_000,
account=FanOut(Zipf()),
created_at=After("account.signed_up_at", within=timedelta(days=365)),
)
The gap is spread uniformly across within using this column's own draw,
so it is a real spread rather than a fixed offset, and it is reproducible
from the seed like everything else.
Two things worth knowing before reaching for it. The result is not
monotonic with the row, because the parents are not: a column filled this
way has a low pg_stats.correlation where
:class:~django_data_shape.distributions.sequential.Sequential gives a high
one. That is honest -- real children of scattered parents arrive scattered --
but it is a different physical shape, and an index scan is costed
differently over it. And the fan-out it reads through may not have a null
share, because a child with no parent has no value to be after; that is
refused when the table is declared rather than discovered as a None in
the arithmetic.
within and at_least are in the column's own units: timedelta for
a datetime column, a number for a numeric one. Anything supporting
parent + offset and offset * float works, which is what makes this
one class rather than a datetime one and a numeric one.
Given ¶
A different distribution per value of parent.field.
Conditional skew, and the reason it is worth declaring: a free account's tickets are mostly closed and an enterprise account's mostly open, so a query filtering on both the plan and the status matches far more or far fewer rows than the product of the two marginals suggests.
Table(
Ticket,
rows=2_000_000,
account=FanOut(Zipf()),
severity=Given(
"account.plan",
{
"free": Skew({"low": 0.9, "high": 0.1}),
"enterprise": Skew({"low": 0.4, "high": 0.6}),
},
default=Skew({"low": 0.7, "high": 0.3}),
),
)
Worth being honest about what this buys. Postgres's own
CREATE STATISTICS cannot span tables, so the planner still estimates
this pair as independent. Declaring it does not fix an estimate; it builds
the database in which the wrong estimate is reproducible, which is the
difference between knowing a query is mis-planned and being told so.
default covers the parent values that were not listed. Without one, an
unlisted value is refused during the load, naming the column and the
value -- one of the very few refusals in this package that cannot happen at
declaration time, because the parent's values live in the parent table and
not in the declaration. Passing a default is how a declaration says it meant
to cover the rest.
canonical ¶
The source, every case in declaration order, and the default.
See Canonical. The cases are ordered for the same reason a Skew's
weights are: each one is a distribution whose own order decides values,
and a mapping this package reordered would be a mapping it had changed.
Aligned ¶
distribution read at a rank shared with every column naming it.
Independent marginals produce a database that is realistic per column and unrealistic per entity: the biggest accounts are not the ones with the most tickets, the most storage or the longest history, because each of those was drawn on its own. No single row is extreme in two ways at once -- and that row is the one that breaks production, and the one a performance test is supposed to find.
A rank is a name the declaration invents. Every column declaring the same rank reads the same draw, so their orderings agree exactly:
Table(
Account,
rows=50_000,
storage_bytes=Aligned("size", Uniform(1e6, 1e12)),
seat_count=Aligned("size", Zipf(1.1)),
trial_days_left=Aligned("size", Uniform(0, 30), reverse=True),
)
reverse=True reads the same rank from the other end, which is how a
column that is inversely related to the others is said. The coupling is
exact in both directions and has no strength parameter: a partial coupling
is a copula, and a copula is a research project wearing a small API. Exact
or reversed covers the shape this exists for, and a declaration that needs
something in between is better served by
:class:~django_data_shape.derivations.derived.Derived over a rank source,
which can compute whatever it likes from the same draw.
Ranks are per table. Two tables using the name "size" share nothing,
because the only thing they could align on is the row index, and row 40 of
one table has no relationship to row 40 of another.
One thing this cannot do for you: a distribution that ignores its draw
aligns to nothing. :class:~django_data_shape.distributions.sequential.Sequential
is a function of the row index and
:class:~django_data_shape.distributions.constant.Constant of neither, so
wrapping either in an Aligned is accepted and does nothing. It is not
refused because a distribution declares no such thing about itself, and
guessing from the type would refuse a caller's own perfectly good one.
canonical ¶
The rank, the distribution read at it, and the direction. See Canonical.
Product ¶
left * right, read from two other columns of this row.
One of three derivations that exist because
:class:~django_data_shape.derivations.derived.Derived -- the only shipped
face that can read another column of the same row -- takes a callable, and
a callable cannot be digested. A shape holding one is refused by
:func:~django_data_shape.template_database.template_database, so a column
as ordinary as total = quantity * unit_price excluded the whole
declaration from the reuse that turns a forty-second build into a
hundred-millisecond clone.
That refusal is right and stays: two lambdas share a name, and identical
bytecode returns something else when a constant it reads is edited in
another module. What was wrong is what it excluded. These three say the
commonest arithmetic as data, so they implement
:class:~django_data_shape.canonical.Canonical and the shape hashes::
Table(
Order,
rows=2_000_000,
quantity=Aligned("basket", Uniform(1, 8, places=0)),
unit_price=Aligned("basket", Uniform(1500, 25000, places=0)),
total=Product("quantity", "unit_price"),
)
Derived is unchanged and remains the answer for computation that really
is code. The line between them is whether the declaration can be written
down: a product of two named columns can, a lambda cannot.
Offset ¶
source + by, where source is another column of this row.
The same-row half of :class:~django_data_shape.derivations.after.After,
which is parent-scoped only. A show goes on sale and then happens; an
invoice is issued and then falls due. Both columns are on the same row, and
until this existed the only way to say so was a lambda -- which cost the
whole shape its template-database cache. See
:class:~django_data_shape.derivations.product.Product for why that
mattered.
The gap is fixed, which is the difference from After: that one
spreads a gap across within using the column's own draw, because a
child's distance from its parent is a real distribution. A due date thirty
days after an issue date is not a distribution, it is a term. Where the
spread is wanted on the same row, Derived still takes a callable.
by is in the column's own units -- a timedelta for a datetime
column, a number for a numeric one -- and anything supporting
source + by works.
Copied ¶
relation.field, unchanged.
A ticket's face value is the unit price of the order it belongs to; a line's
currency is its invoice's. There is no arithmetic at all, which is what made
needing a lambda for it -- and losing the template-database cache with it --
annoying rather than merely unfortunate. See
:class:~django_data_shape.derivations.product.Product.
Worth saying what this is not, because the two are easy to confuse. A denormalised copy is a column with its own statistics: the planner sees a distribution over the child table rather than a join, which is the whole reason schemas carry such columns and the reason a shaped database has to reproduce them. Reading the parent's column through the join is a different query with a different plan.
Keys¶
KeyStrategy ¶
Bases: Protocol
Turns a row index into that row's primary key.
The generalisation of what used to be a hard-coded dense 1..N range. The
range was never the requirement: what the design actually rests on is that
the key is a deterministic function of the row index, and integers were
only the most obvious such function.
Everything the dense range bought is bought by determinism instead. A child
can compute its parent's key from the parent's index, so a foreign key is
satisfied without a lookup whatever the key type. A self-referential tree is
acyclic because parent_index < child_index holds on the index, not on the
value. And two builds of one shape agree because the same seed produces the
same keys.
stream is a per-table value derived from the seed, so a strategy that
needs entropy has some. One that does not -- a counter, or a caller's own
function -- ignores it, exactly as a positional distribution ignores its
draw.
SequentialKeys ¶
row + 1: the default for any integer primary key.
Counting from one rather than zero because that is what a database sequence does, and a test database whose keys start at zero is subtly unlike every other one the reader has seen.
This is the strategy that obliges the sequence reset after loading. It is also the only one that does: a key type with no sequence behind it has nothing to move.
It is also the only strategy in this package that can say itself in SQL, so
it is the only one that can fill a
:class:~django_data_shape.projection.Projection -- see
:class:~django_data_shape.keys.sql_keys.SqlKeys. That is not an accident
of implementation effort: row + 1 is arithmetic every database has, and
a keyed hash is not.
key_sql ¶
The same row + 1, for a row index the database computes.
Written as the Python expression rather than as row_number()
directly, so the two halves of this strategy stay visibly the same rule.
The caller decides what a row index is in SQL; this only says what the
key is, given one.
canonical ¶
Nothing to say: the rule is row + 1 and has no parameters. See Canonical.
The empty tuple is not the same as being absent -- the strategy's own
type name is part of the digest, so this differs from UuidKeys()
below, which also has nothing to say.
UuidKeys ¶
A UUID per row, deterministic in the seed and the row index.
Derived from a hash rather than drawn from uuid4 for the reason the
whole package is built around: two builds of one shape have to agree, and a
random key would make the primary key -- and therefore every foreign key
pointing at it -- differ between runs.
A full 128 bits from the digest, not a float draw. A draw carries 53 bits, which sounds ample until birthday collisions arrive around ninety million rows; a table that large is exactly the kind this package exists to build.
The version and variant bits are stamped so the result is a well-formed version 4 UUID. Applications store v4 keys, so a test database holding something that merely looks UUID-shaped would be unlike the thing it stands in for.
is_disjoint_from_existing_rows ¶
Always. A 128-bit digest cannot land on a caller's row. See Disjoint.
The full digest is what makes this a statement rather than a hope. A draw carries 53 bits, where birthday collisions arrive around ninety million rows; these are 122 bits after the version and variant are stamped, which is the space a version 4 UUID has and the space every application storing one is already relying on.
canonical ¶
Nothing to say: the digest is a function of the seed and the row. See Canonical.
Md5Keys ¶
A version 4 shaped UUID per row, derived in Python and in SQL.
The projection half of :class:~django_data_shape.keys.uuid_keys.UuidKeys,
and a separate strategy rather than a second meaning for that one. The two
produce different keys for the same row, so quietly making one become the
other where a SQL twin is needed would change every key in every world
already built, and would give one declaration two meanings depending on
which statement filled the table.
Why md5 rather than blake2b. A
:class:~django_data_shape.projection.Projection has no declared row count,
so its rows never pass through Python and its keys have to be assigned by
the INSERT ... SELECT that writes them -- which means the hash has to
exist on both sides and agree byte for byte. blake2b has no PostgreSQL
equivalent; md5 is in the standard library and is a built-in function of
the server. That is the whole of the reason, and it is worth being plain
that this is not a security choice: nothing here authenticates anything,
the input is a table's own seed and row index, and md5's weakness is
collision resistance against an adversary who chooses the input. Nobody
chooses these inputs.
128 bits, of which 122 survive the version and variant stamp -- the same space every application storing a v4 UUID already relies on, and far past where birthday collisions matter for a test database.
usedforsecurity=False is passed because it has to be: on a FIPS build,
hashlib.md5 without it raises rather than returning a digest, which
would make this strategy unusable on hosts that are otherwise fine.
key_sql ¶
The same digest, stamped the same way, computed by the server.
The two halves are checked against each other rather than argued about: a test computes both for the same rows and compares them, which is the only form of agreement that means anything here.
md5 takes the same eight-byte big-endian pair Python hashes, turned
back into bytes by decode, so neither side is hashing a rendering of
the other's input. The stamp is two overlay calls on the hex digest:
nibble 13 is the version and is always 4, and nibble 17 is the
variant, which keeps its low two bits and takes 8 in the high two.
The stream is embedded as hex rather than converted by the server,
and that is a fix rather than a preference. It used to be written
to_hex(<stream>::bigint), which asks PostgreSQL to re-derive a
number Python produced as unsigned -- and bigint is signed, so any
stream above 2^63 raised NumericValueOutOfRange before a row was
written. That is a coin flip per table, not a property of any schema:
the stream is a hash of the table and field names, so roughly half of
all table names land above the limit. It is a constant by the time this
statement is built, so nothing about it needs deriving at all.
row stays a conversion because it is genuinely an expression the
server evaluates, and a row index is far below the limit.
is_disjoint_from_existing_rows ¶
Always, for the reason UuidKeys gives. See Disjoint.
canonical ¶
Nothing to say: the digest is a function of the seed and the row. See Canonical.
Empty like UuidKeys' own, and not the same thing -- the strategy's
type name is part of the digest, so two shapes differing only in which
of them they use are two different worlds, which they are.
KeyFunction ¶
A caller's own deterministic mapping from row index to key.
The escape hatch for a key this package cannot infer -- a natural key, a prefixed slug, an external identifier. Integer and UUID primary keys are inferred and need none of this; anything else is declared rather than guessed, because a guessed value in a semantic column is how a character primary key once got loaded with the strings "1", "2" and "3".
The function must be a pure function of the row index. That is checked on a sample at construction rather than trusted: a key that varies between calls would break reproducibility, and it would break it silently, in the one column every foreign key points at.
SqlKeys ¶
Bases: Protocol
A :class:~django_data_shape.keys.key_strategy.KeyStrategy with a SQL twin.
Every table in a shape has a declared row count, so its keys can be
enumerated in Python and streamed into COPY. A
:class:~django_data_shape.projection.Projection has no declared row count
-- its cardinality is determined by the join it copies along -- so there is
no range of row indices to enumerate, and the rows never pass through Python
at all. The keys have to be assigned by the statement that inserts them.
That is the whole reason this protocol exists, and it is deliberately an extension of the key strategy rather than a second way to decide keys. A projected table's keys come from the same place as every other table's: the strategy on the declaration. The only extra requirement is that the strategy can express the same rule as an expression the database evaluates.
key_sql mirrors
:meth:~django_data_shape.keys.key_strategy.KeyStrategy.key_for argument
for argument. stream is the same per-table value derived from the seed.
row is a SQL expression that evaluates to the same zero-based row index
key_for receives, so an implementation writes the same arithmetic it
would write in Python -- SequentialKeys returns (row) + 1 for
exactly the reason its Python half returns row + 1.
A strategy that does not implement this is not broken and is not second-class; it simply cannot fill a projected table, and it is refused by name when one is declared over it. Approximating instead -- a different hash in SQL from the one Python computes -- would give one strategy two meanings depending on which statement filled the table, which is the quiet divergence this package exists to prevent.
SqlValue ¶
One projected column's value, as SQL over the join the projection derives.
A projection copies a column from the source it names, or takes the model's own default, and those are the only two answers it has. A projected table's measure column is neither: the score on a review, the amount on a generated line, the reading on a sample. It belongs to the projected row and to nothing the source carries.
Leaving it to the model default is legal and is the wrong answer for this
package specifically: one value across every projected row is
n_distinct = 1, which is the exact shape a planner cannot use. A library
whose whole purpose is planner realism would then be building a table it had
made unplannable, and the declaration would look correct.
sql= already answers this and answers it expensively: it replaces the
whole SELECT, so the join stops being derived from the model graph and
can drift from it afterwards, the copied columns are written out by hand,
and the key strategy has to be spelled in SQL. values= gives up none of
that and writes one expression for the one column that needs one::
Projection(
ReviewScore,
per=Review,
copying=Criterion,
values={"score": SqlValue("({per}.id * 31 + {source}.id * 17) % 5 + 1")},
)
{per} and {source} are substituted with the aliases the derived
statement uses, quoted for the connection. They are placeholders rather than
the aliases themselves because the aliases are this package's private
business: a declaration that spelled them would break the day they changed,
and a reader could not tell which side was which.
On a UUID-keyed schema the example above is a type error, and it is worth
saying so here rather than leaving it to the build. A project that gives
every model id = UUIDField(primary_key=True) from one abstract base is an
ordinary Django layout, and uuid * integer has no operator in PostgreSQL:
the expression is opaque to this package, so the refusal comes from the
driver, at build time, naming neither the shape nor the column. The variation
has to come from a hash instead::
values={
"score": SqlValue(
"abs(hashtext({per}.id::text || {source}.id::text)::bigint) % 5 + 1"
)
}
Each part of that earns its place. ::bigint goes before abs
because hashtext returns int4 and abs(-2147483648) is integer
out of range; abs is there at all because PostgreSQL's % keeps the
sign of the dividend, so hashtext(...) % 5 spans negatives and a measure
column would quietly hold them.
An expression may name another column in the same values=, as
{values.<name>}. A projected table's measure columns are usually related
to each other -- a requested amount and an approved one, a quantity and a
total -- and without this the relationship has to be restated from whatever
both were computed from, with coefficients chosen so that it happens to
hold::
values={
"requested_amount": SqlValue(
"abs(hashtext({per}.id::text)::bigint) % 500 + 100"
),
"approved_amount": SqlValue("{values.requested_amount} * 8 / 10"),
}
The name is dotted rather than a bare {requested_amount} because
{per} and {source} already occupy that space and a model is entitled
to a column called per. Only values= entries are referenceable: a
copied column is already reachable as {source}.name, and the primary key
is the row_number() window itself and is reachable nowhere. A name that
resolves to neither, and a cycle, are refused at declaration time.
It is substitution, not sharing, and that spelling is deliberate:
{values.x} names the declaration, and what it splices in is that
expression written out again. The database evaluates it once per reference.
For a deterministic expression -- which every expression here has to be
anyway, since
:func:~django_data_shape.template_database.template_database reuses a
database keyed on the declaration and nothing else -- that costs arithmetic
nobody measures. For a volatile one the two copies are two different values,
and the relationship the declaration appears to state is not the one the rows
hold. State it in the declaration and net it with an
:class:~django_data_shape.invariant.Invariant, which is what catches that
case and every other way the rule can stop being true.
It is SQL rather than a distribution, and that is a decision worth
stating. A :class:~django_data_shape.distributions.distribution.Distribution
computes from draw(stream, row), which is SplitMix64 -- expressible in
PostgreSQL only through numeric modular arithmetic and casts across the
sign boundary, where a single mistake gives one declaration two meanings
depending on which statement filled the table. That is the divergence
:class:~django_data_shape.keys.sql_keys.SqlKeys exists to refuse, and it
is not worth buying convenience with. An expression the caller wrote is
honest about being the caller's.
It is the one part of a shape that is not portable, and it cannot be
made portable. Everything else here is a declaration this package
compiles per backend; an expression is SQL the database evaluates as
written. mod(x, 5) returns an integer on PostgreSQL and a REAL on
SQLite, so one declaration writes 5 into one database and 5.0 into
the other -- which is why the example above uses %, integer on both.
Nothing here can detect that: the expression is opaque until the database
reads it. Write it for the backend the shape is built on, and cast when it
has to be both.
% is passed through as the operator, which takes an escape this package
applies rather than asking for -- unlike sql=, which takes params=
and so owns its own placeholders; a lone % is refused there instead. The statement is executed with bound
parameters, so a literal % in it is an incomplete placeholder to both
psycopg and Django's SQLite wrapper, and the declaration fails at execution
with a paramstyle error naming nothing in the shape. The paramstyle is this
package's private business for the same reason the join's aliases are: a
declaration that spelled it would be spelling how the statement happens to
be run.
The expression is inert data, so a shape holding one still digests.
render ¶
The expression with the join's aliases substituted and % escaped.
The escape is last so it also covers a % inside a quoted alias,
and it is unconditional because the statement always carries a
parameter sequence -- an empty one is still a sequence, and both
drivers interpolate on anything that is not None.
canonical ¶
The expression itself, which is what decides the rows. See Canonical.
Disjoint ¶
Bases: Protocol
A key strategy that cannot produce a key some existing row already holds.
The opt-in protocol behind one refusal, and the refusal is worth stating
before the protocol is. Building into a table that already has rows is
normally refused, because this package assigns keys from 1 every time and a
second build collides on the primary key -- an IntegrityError from
inside COPY naming an index, which tells a reader nothing.
That reasoning is about integer keys, and the refusal was not. A
:class:~django_data_shape.keys.uuid_keys.UuidKeys table derives a
128-bit digest per row and cannot collide with anything a caller's factory
wrote, so the refusal blocked the hybrid the documentation advertises --
parents made by your own code, children made by this package -- for exactly
the schemas where UUID keys are the norm.
A strategy that does not implement this is read as not disjoint, which is
the safe direction and the reason it is opt-in rather than a flag with a
default. :class:~django_data_shape.keys.key_function.KeyFunction is the
case that decides it: the caller's own function could return anything, this
package cannot read it, and guessing "probably fine" would trade a clear
refusal for a load that dies partway through.
is_disjoint_from_existing_rows returns a bool rather than the protocol
being a bare marker, for the reason
:class:~django_data_shape.distributions.distinct.Distinct does: the answer
can be a property of the parameters and not only of the class.
It says nothing about anything but keys. A unique constraint on some other column can still collide with a row that was already there, and a business invariant can still be broken by rows this package did not write. Both are checked after the load, against the table as it then stands, which is the reading that stays true either way.
Distributions¶
Distribution ¶
Bases: Protocol
Produces the value of one field for one row.
Both arguments are supplied to every implementation because the two kinds of
distribution need different halves: a categorical or numeric one consumes
draw and ignores the row, while a monotonic one consumes row and
ignores the draw. Passing both keeps the protocol single-method, and a
single-method protocol is what allows a distribution to be a plain object
rather than a class hierarchy.
draw is uniform in [0, 1) and depends only on the field and the row, so
an implementation must not carry state between calls. One that did would
make the same shape produce different data depending on generation order,
which is the property the placement work in a later release depends on.
Bounded ¶
Bases: Protocol
A distribution with a known, finite number of distinct values.
Deliberately a second protocol rather than a method on
:class:~django_data_shape.distributions.distribution.Distribution. Adding
it there would make it required, and a custom distribution written against
the single-method protocol would stop satisfying it -- so the one thing this
exists to prevent, a declaration that cannot describe a database, would be
bought by breaking every declaration someone had already written.
Structural and runtime-checkable, so a distribution opts in by having the method and nothing has to register anywhere. A distribution that cannot answer -- one drawing from a continuous range, say -- simply does not implement it, and is treated as unbounded rather than as suspicious.
Categorical ¶
Bases: Protocol
A distribution over a known set of values, with a known share for each.
The fourth opt-in protocol, beside
:class:~django_data_shape.distributions.bounded.Bounded,
:class:~django_data_shape.canonical.Canonical and
:class:~django_data_shape.keys.sql_keys.SqlKeys, and added for the one
question a business invariant asks that none of the others can answer:
how many of these rows will carry this particular value?
That question is what turns a partial UniqueConstraint from an error
message at row 700,000 into arithmetic at declaration time.
one_active_project_per_company permits one row per company with
status='ACTIVE'; a Skew giving ACTIVE a tenth of two million
rows asks for two hundred thousand of them. Both numbers are known before a
row is generated, and only shares supplies the second.
It is deliberately not the same protocol as Bounded, although
:class:~django_data_shape.distributions.skew.Skew and
:class:~django_data_shape.distributions.constant.Constant implement both.
Bounded answers how many different values can this produce, which is a
question about capacity and is answerable by a distribution that could never
enumerate itself -- a shuffled range of ten thousand integers, say.
Categorical answers which values, and in what proportion, which is a
question about content. A distribution that can answer the second can always
answer the first; the reverse is not true, so joining them would have made
the cheap claim cost the expensive one.
shares returns each value mapped to its share of the rows, summing to
one. The shares are the declaration's own arithmetic rather than a
measurement: what comes back is what the declaration asked for, which is
exactly what a refusal should quote back at it.
A distribution that cannot enumerate itself simply does not implement this, and is treated as undecidable rather than as suspicious -- the constraint it might have broken is then left to the post-load check and to the database.
Ascending ¶
Bases: Protocol
A distribution that can say whether its values rise with the row index.
The fifth opt-in protocol, and the narrowest. It exists for one parameter:
:class:~django_data_shape.derivations.per_parent.PerParent's order_by,
which claims that the last row of a group under this column's ordering is
the last row of the group as the fan-out partitioned it. That claim is true
only if the column climbs with the row index, and only a distribution can
say whether it does.
is_ascending returns a bool rather than the protocol being a bare
marker, because the answer is a property of the parameters and not of the
class. :class:~django_data_shape.distributions.sequential.Sequential with
a positive step climbs and with a negative step falls, and a declaration
that asked for the newest row of each group while filling the column
backwards would get the oldest -- silently, and in exactly the data nobody
inspects by hand.
Structural and opt-in for the reason every protocol here is: inferring the
answer from the type would refuse a caller's own perfectly monotonic
distribution, and asserting it on every Distribution would break the
ones already written against the single-method protocol.
Distinct ¶
Bases: Protocol
A distribution whose value differs in every row of the table.
The sixth opt-in protocol, and the exact dual of
:class:~django_data_shape.distributions.bounded.Bounded: that one says
how few different values can this produce, this one says it produces a
different one every time.
It exists for one question, and the question is not about capacity.
A multi-column UniqueConstraint needs the tuple to be distinct, and
every mechanism this package has for filling a column -- a
:class:~django_data_shape.fan_out.FanOut partition, and every
:class:~django_data_shape.distributions.distribution.Distribution --
computes its column from the row index and from nothing else. So no column
can see what another column put in the same row, nothing enumerates the
tuples, and whether two rows collide is a matter of the seed. That is
refused by
:func:~django_data_shape.check_constraints.check_constraints rather than
left to fail inside COPY.
One kind of column keeps such a constraint anyway, and keeps it without
coordinating with anything: one whose own values are already distinct. A
pair is distinct as soon as either half is, so (company, invoice_number)
is unique for free when invoice_number is. Distinct is how a
distribution says that about itself, and saying it is what separates the
declaration that builds from the one that is a lottery.
is_distinct_per_row returns a bool rather than the protocol being a bare
marker, for the same reason
:class:~django_data_shape.distributions.ascending.Ascending does: the
answer is a property of the parameters and not of the class.
:class:~django_data_shape.distributions.sequential.Sequential with a
non-zero step writes a different value in every row and with a zero step
writes one value in all of them, and those are the same class.
Structural and opt-in like every protocol here. A distribution that does not implement it is read as not distinct, which is the safe direction: the worst that costs is a refusal a caller answers by adding one method, where the other reading costs a load that dies at a row number which moves when the seed does.
Skew ¶
Values drawn from a weighted set, in a fixed order.
This is the distribution the package exists for. A status column that is 98% one value is what decides whether an index on it is usable at all, and it is the thing a fixtures loop never expresses -- ten rows with one of each says the opposite of what production says.
Mapping[Any, float] rather than dict[object, float]: dict is
invariant in both parameters, so a caller's prepared dict[str, float] --
the obvious way to build one of these outside a call -- would be rejected by
a type checker for no reason a reader could act on. Mapping is covariant
in its value type, so integer counts are accepted too.
Weights are relative and need not sum to 1: the readable form is often counts, and normalising here is cheaper than making every caller do it. They must be positive, because a zero-weight value is a value that never appears, which is better said by leaving it out than by declaring it and meaning not.
shares ¶
Each declared value against its share of the rows. See Categorical.
The normalised weights rather than the raw ones, because the question a
caller of this asks is "how many of my rows will hold this value", and
weights are relative by design -- counts are a legitimate way to write
one. Normalising here is what lets a refusal quote a row count back at a
declaration that only ever said 0.1.
canonical ¶
The weights, in declaration order. See Canonical.
The order is part of the declaration rather than a detail of how it was written: the cumulative bounds above are laid out in it, so the same weights given in a different order hand a different value to the same draw. Sorting here would tell the template cache that two databases which differ row for row are the same one.
Uniform ¶
Values spread evenly between low and high.
Deliberately the least interesting distribution in the package, and named plainly so it reads as a choice. Most real columns are not uniform, and a uniform declaration on a column that matters is usually a placeholder somebody meant to come back to.
places rounds the result. Not because the column would reject the
unrounded value -- Postgres rounds a float to a numeric(10, 2) happily,
and only overflowing the declared precision is an error -- but because a
money column whose values carry full binary float noise is not what the
application would ever have written, and this package's whole claim is that
the loaded rows are ones it could have. Rounding to Decimal rather than
float keeps the value exact on the way into COPY; places=0 is
how a plain integer column is filled.
canonical ¶
The bounds and the rounding. See Canonical.
The precision computed in __init__ is left out, because it is a
function of these three and adding it could only ever agree with them.
Sequential ¶
start plus row steps: monotonic, and correlated with the key.
The point is the correlation, not the convenience. Postgres records a correlation statistic per column and costs an index scan differently depending on it, so a timestamp column filled with shuffled dates plans differently from one that advances the way real rows arrive. Shuffling is the easy thing to do by accident and it is wrong in a way that only shows up in plan choice.
Works for anything supporting start + row * step, which covers numbers
and datetime with a timedelta. It ignores draw entirely: this is
the one distribution whose value is a function of position alone.
is_distinct_per_row ¶
Whether every row gets its own value. See Distinct.
True for any non-zero step, in either direction: this is a statement
about injectivity and not about order, so a column counting backwards
keeps a unique constraint exactly as well as one counting forwards.
Ascending is deliberately not reused for it -- that protocol answers
which end of a group is last, and a caller's own distribution may
climb without strictly climbing, which would make it an unsound proof of
distinctness.
self._step * 0 rather than a literal zero, for the reason
:meth:is_ascending computes its own: a date column steps by a
timedelta and a numeric one by a number.
is_ascending ¶
Whether the step moves values up rather than down. See Ascending.
self._step * 0 rather than a literal zero, for the reason
:class:~django_data_shape.derivations.after.After computes its own: a
date column steps by a timedelta and a numeric one by a number, and
this is the only spelling of zero that does not have to ask which.
Zipf ¶
Positive weights following a power law of exponent s.
The distribution fan-out is realistically drawn from, and the reason
declaring fan-out is worth doing at all. A customer table where every
customer has ten orders is not merely tidy, it is the one shape in which the
planner is never wrong: its n_distinct average is the truth, so a join
estimate cannot miss. Give the head a thousand orders and the tail one, and
the same estimate is out by orders of magnitude in both directions -- which
is what production looks like and what a test database has to reproduce.
Inverse transform of a Pareto: (1 - draw) ** (-1 / s). Larger s
means a lighter tail; values near 1 are the classic Zipf regime.
Constant ¶
Every row gets value.
Present because a column that never varies is a real shape, not a missing declaration: a tenant id on a single-tenant fixture, or a flag that is false for every row in the dataset under test. Declaring it says so, where leaving it out would mean the field simply had no distribution.
It also has a planner consequence worth knowing: a single-valued column has exactly one most-common value at frequency 1.0, so a filter on it is either everything or nothing, and an index on it is never usable.
shares ¶
The one value, holding all of the rows. See Categorical.
Worth implementing rather than leaving undecidable, because a constant
is how this package fills a column the model defaults -- so a status
column with default="COMPLETE" and no declaration of its own arrives
here, and a constraint conditioned on ACTIVE is then satisfied by a
table that never writes one. Undecidable would refuse it.
Errors¶
InvalidShape ¶
Bases: Exception
A shape declaration is contradictory, incomplete or unsatisfiable.
Its own type, and raised as early as the contradiction can be seen -- at declaration time wherever possible, rather than at load time. The reason is the package's own bar: a generated database that is wrong is worse than one that refuses to exist, because the test suite it feeds will assert on data that could never occur and pass or fail for reasons unrelated to the code.
Every message names the model, the field or the constraint at fault. An error that says only that something is inconsistent leaves the reader to re-derive what this code already knew.
UnsupportedBackend ¶
Bases: Exception
The database backend cannot support the operation requested.
Separate from :class:~django_data_shape.invalid_shape.InvalidShape
because nothing is wrong with the declaration: the same shape is valid, and
would build, against Postgres. Only the destination is unsuitable.
Raised rather than warned, and never quietly degraded to a slower path. The
whole claim of this package is that the loaded database is one the planner
can reason about; a backend without COPY or column statistics cannot
produce that, and silently producing something else would be the failure
mode the package was written to expose.
DerivationQueriedDatabase ¶
Bases: Exception
Callable code supplied to a shape queried the database while generating.
Its own type because the rule it enforces is this package's boundary rather than a detail of one declaration: this package may call your code, but your code may not call the database.
The most likely feature request this package will ever receive is a per-row
creation hook -- "call my service to build each object" -- and it is
declined permanently. Model.objects.create() per row is the thing being
replaced; offering it makes it the default path, because it is the easiest
thing to write, and a package whose default path is not COPY has no
reason to exist.
What makes the refusal worth stating as a rule rather than as advice is that it is decidable. The generation pass runs under a wrapper on the connection being built, so a query raises this rather than quietly costing a round trip per row -- a fact rather than a convention.
The check sees queries on the connection being built. Code that reaches a different alias, or another thread's connection, is outside what a wrapper on one connection can observe; the rule still holds there, and only its enforcement stops at the edge of the connection.
ShapeNotEmpty ¶
Bases: Exception
The destination table is not empty, so the assigned keys would collide.
Its own type rather than a reused one because the caller's remedy is specific and nothing else in this package shares it: empty the table, then build. Raised before any row is written, so a build that fails this way has changed nothing.
The alternative was letting the database report it, which it did -- as a unique-violation naming an index. That says what went wrong at the storage layer and nothing about what the caller did or what to do instead.
UnhashableShape ¶
Bases: Exception
A shape holds something whose contribution to the data cannot be read.
Its own type rather than :class:~django_data_shape.invalid_shape.InvalidShape
because nothing is wrong with the declaration: it builds, it is reproducible,
and every row it produces is correct. Only one thing cannot be done with it,
and that is deciding whether a database built from it earlier is a database
built from this one.
Raised rather than answered with a digest that leaves the unreadable part out. That is the whole point: a cache key which ignores something is a key that stays the same when the data changes, and the failure it produces is a test suite running against a database nobody asked for -- silently, and in the direction that looks like everything is working. A refusal costs a build.
InvariantViolated ¶
Bases: Exception
A declared invariant found rows that should not exist.
Its own type rather than an :class:~django_data_shape.invalid_shape.InvalidShape,
because it is a different kind of wrong. An invalid shape is a declaration
that could not describe any database; a violated invariant is a declaration
that described one and then did not build it. The first is answered by
rewriting the declaration, the second by asking which of the two -- the rule
or the generator -- is lying.
It fails the build, and it does so inside the transaction that loaded the rows, so nothing lands. That is the more useful of the two readings: an invariant that failed the test would leave a database full of impossible data for every later assertion to be evaluated against, and those assertions would pass or fail for reasons unrelated to the code. A build that refuses leaves the database exactly as it was found.
The message carries the rule's name, how many rows broke it and a sample of them, because a build failure is read out of a terminal rather than stepped through in a debugger -- and a rule that only says it was violated has handed the reader back the work it just did.
WorldChanged ¶
Bases: Exception
The rows a question is being asked about are not the rows that were built.
Its own type because it is the opposite failure from
:class:~django_data_shape.invalid_shape.InvalidShape: the declaration is
fine and the answer would be arithmetically correct. It would simply be an
answer about a different database from the one the caller is looking at.
Raised by :func:~django_data_shape.fan_out_sizes.fan_out_sizes, which
recomputes a fan-out's partition rather than aggregating the child table.
That recomputation takes one thing from the database -- the parent keys --
so a parent table that has gained or lost rows since the build produces a
partition that never existed. Every number would look plausible and every
one of them would be wrong, which is precisely the class of failure this
package refuses to ship: a shaped database that quietly means something
other than what it says.