Write a prioritization rule¶
A prioritization rule is a predicate: a condition on a variant's columns, such as "|logfc| is at least 0.5 and the
variant is in a peak". A model binding uses one to decide which variants are prioritized, and so can an annotation
source. altar.predicates builds the rule once as a Predicate object. Altar renders the same object as a Python
function, which a local store runs over its rows, and as a SQL expression, which a warehouse store runs in its own
query.
altar.predicates is stable, except SqlDialect, which is
provisional.
Build a predicate¶
Terms produce values; predicates compare and combine them into a boolean.
from altar.predicates import Col, Lit, Abs, Ge, Gt, Eq, And, Or, Not
# "a large effect in an accessible region": |logfc| >= 0.5 AND in_peak = TRUE
predicate = And(
Ge(Abs(Col("logfc")), Lit(0.5)),
Eq(Col("in_peak"), Lit(True)),
)
| Piece | Options |
|---|---|
| Terms | Col(name) (a column), Lit(value) (a constant), Abs(term) (absolute value), Greatest(*terms) and Least(*terms) (null-ignoring maximum and minimum) |
| Comparisons | Ge, Gt, Le, Lt, Eq, Ne, each taking two terms |
| Tests | IsNull(term), In(term, values) (membership in a tuple of literals) |
| Combinators | And(*preds), Or(*preds), Not(pred) |
Render it as Python or SQL¶
The same object renders both ways:
predicate.to_python() # -> callable: row(dict) -> bool
predicate.to_sql() # -> "(ABS(logfc) >= 0.5 AND in_peak = TRUE)"
fn = predicate.to_python()
fn({"logfc": -0.8, "in_peak": True}) # True
to_sql takes an optional first argument, col, a function from a column name to the SQL that reads it. The
BigQuery store passes one that qualifies each name with its table alias.
to_sql also takes a dialect, which sets how a string literal is escaped. The default,
SqlDialect.STANDARD, is for SQLite and DuckDB. Pass SqlDialect.BIGQUERY for SQL that BigQuery runs:
from altar.predicates import Col, Eq, Lit, SqlDialect
Eq(Col("trait"), Lit("Alzheimer's")).to_sql() # -> "trait = 'Alzheimer''s'"
Eq(Col("trait"), Lit("Alzheimer's")).to_sql(dialect=SqlDialect.BIGQUERY) # -> "trait = 'Alzheimer\\'s'"
Handle missing values¶
Predicates follow SQL's three-valued (Kleene) logic, and both renderings treat missing values the same way. A comparison
that reads a missing or None value is unknown (None), not False. An unknown value passes through And, Or and
Not as SQL NULL does. Only a definite True is a match:
Ge(Col("logfc"), Lit(0.5)).evaluate({"logfc": None}) # None (unknown — not False)
predicate.to_python()({"logfc": None}) # False (to_python matches only definite True)
Compare a nullable boolean column explicitly, as Eq(Col("in_peak"), Lit(True)), which renders to
in_peak = TRUE. Both renderings then handle a null the same way, and a variant is prioritized in a local store
exactly when it is in the warehouse.
Test for nulls, membership and extremes¶
IsNull(term) is the one test that is never unknown: it is True for a missing or None value and False
otherwise, and renders to (term IS NULL). Write its negation as Not(IsNull(term)). Use it to say what a
missing value should mean instead of letting it silently fail to match:
from altar.predicates import Col, In, IsNull, Or
# "a promoter or enhancer variant, or one the region annotation has not covered"
predicate = Or(In(Col("region_type"), ("promoter", "enhancer")), IsNull(Col("region_type")))
predicate.to_sql() # -> "((region_type IN ('promoter', 'enhancer')) OR (region_type IS NULL))"
In(term, values) follows SQL IN: it is True when the term equals one of the values, unknown when the term
is null, and False otherwise. The values are literals of one type family (booleans, numbers, or strings).
None is rejected, because SQL makes x IN (..., NULL) unknown rather than true for a null x; combine
In with IsNull as above to match nulls. An empty tuple is False for every row and renders to FALSE,
since IN () is not valid SQL.
Greatest(*terms) and Least(*terms) take the maximum or minimum over the operands that are not null, and
are null only when every operand is. That is the shape of "the highest allele frequency over the ancestry
groups that have one":
from altar.predicates import Col, Ge, Greatest, Lit
common = Ge(Greatest(Col("af_afr"), Col("af_nfe"), Col("af_eas")), Lit(0.01))
common.evaluate({"af_afr": None, "af_nfe": 0.02}) # True: one missing group does not hide the others
BigQuery's GREATEST returns NULL when any argument is NULL, while DuckDB's and PostgreSQL's skip NULL arguments. To
give the same answer on every engine, and the same as the Python rendering, Greatest renders each argument as a
COALESCE of the operands, rotated to start at a different one:
GREATEST(COALESCE(af_afr, af_nfe, af_eas), COALESCE(af_nfe, af_eas, af_afr), COALESCE(af_eas, af_afr, af_nfe))
The SQL text grows with the square of the operand count.
A float NaN is a value, not a null, in both renderings: IsNull is False for it, and Greatest and Least
do not skip it. A source emits None (SQL NULL) for a missing value, not NaN.
In values and Lit constants may be NumPy scalars, for example values taken from an array. They render as
plain SQL literals, and In stores them as built-in Python scalars with duplicates removed in
first-occurrence order. Pass an ordered collection such as a tuple or list; a set is rejected because its
order, and so the SQL text, can differ between processes.
Attach a rule to a model or a source¶
A ModelPlugin returns a Predicate from prioritize_predicate() (Add a model
binding), and a prioritizing AnnotationSource
returns one as well. The SQLite store runs the Python rendering, and BigQueryScoreStore runs the SQL
rendering in its query.
Next¶
- What prioritized means explains how the rules of every model and annotation source combine.
- Tutorial 4: write your own prioritization rule writes, evaluates and renders a rule over an export in a notebook.
- Run and read interpretations explains one model's prediction for a variant, base by base.