Skip to content

Write a prioritization rule

A prioritization rule is a predicate: a condition on a variant's columns, such as "|logfc| is at least 0.5 and the variant is in a peak". A model binding uses one to decide which variants are prioritized, and so can an annotation source. altar.predicates builds the rule once as a Predicate object. Altar renders the same object as a Python function, which a local store runs over its rows, and as a SQL expression, which a warehouse store runs in its own query.

altar.predicates is stable, except SqlDialect, which is provisional.

Build a predicate

Terms produce values; predicates compare and combine them into a boolean.

from altar.predicates import Col, Lit, Abs, Ge, Gt, Eq, And, Or, Not

# "a large effect in an accessible region":  |logfc| >= 0.5  AND  in_peak = TRUE
predicate = And(
    Ge(Abs(Col("logfc")), Lit(0.5)),
    Eq(Col("in_peak"), Lit(True)),
)
Piece Options
Terms Col(name) (a column), Lit(value) (a constant), Abs(term) (absolute value), Greatest(*terms) and Least(*terms) (null-ignoring maximum and minimum)
Comparisons Ge, Gt, Le, Lt, Eq, Ne, each taking two terms
Tests IsNull(term), In(term, values) (membership in a tuple of literals)
Combinators And(*preds), Or(*preds), Not(pred)

Render it as Python or SQL

The same object renders both ways:

predicate.to_python()  # -> callable: row(dict) -> bool
predicate.to_sql()  # -> "(ABS(logfc) >= 0.5 AND in_peak = TRUE)"

fn = predicate.to_python()
fn({"logfc": -0.8, "in_peak": True})  # True

to_sql takes an optional first argument, col, a function from a column name to the SQL that reads it. The BigQuery store passes one that qualifies each name with its table alias.

to_sql also takes a dialect, which sets how a string literal is escaped. The default, SqlDialect.STANDARD, is for SQLite and DuckDB. Pass SqlDialect.BIGQUERY for SQL that BigQuery runs:

from altar.predicates import Col, Eq, Lit, SqlDialect

Eq(Col("trait"), Lit("Alzheimer's")).to_sql()  # -> "trait = 'Alzheimer''s'"
Eq(Col("trait"), Lit("Alzheimer's")).to_sql(dialect=SqlDialect.BIGQUERY)  # -> "trait = 'Alzheimer\\'s'"

Handle missing values

Predicates follow SQL's three-valued (Kleene) logic, and both renderings treat missing values the same way. A comparison that reads a missing or None value is unknown (None), not False. An unknown value passes through And, Or and Not as SQL NULL does. Only a definite True is a match:

Ge(Col("logfc"), Lit(0.5)).evaluate({"logfc": None})  # None  (unknown — not False)
predicate.to_python()({"logfc": None})  # False (to_python matches only definite True)

Compare a nullable boolean column explicitly, as Eq(Col("in_peak"), Lit(True)), which renders to in_peak = TRUE. Both renderings then handle a null the same way, and a variant is prioritized in a local store exactly when it is in the warehouse.

Test for nulls, membership and extremes

IsNull(term) is the one test that is never unknown: it is True for a missing or None value and False otherwise, and renders to (term IS NULL). Write its negation as Not(IsNull(term)). Use it to say what a missing value should mean instead of letting it silently fail to match:

from altar.predicates import Col, In, IsNull, Or

# "a promoter or enhancer variant, or one the region annotation has not covered"
predicate = Or(In(Col("region_type"), ("promoter", "enhancer")), IsNull(Col("region_type")))
predicate.to_sql()  # -> "((region_type IN ('promoter', 'enhancer')) OR (region_type IS NULL))"

In(term, values) follows SQL IN: it is True when the term equals one of the values, unknown when the term is null, and False otherwise. The values are literals of one type family (booleans, numbers, or strings). None is rejected, because SQL makes x IN (..., NULL) unknown rather than true for a null x; combine In with IsNull as above to match nulls. An empty tuple is False for every row and renders to FALSE, since IN () is not valid SQL.

Greatest(*terms) and Least(*terms) take the maximum or minimum over the operands that are not null, and are null only when every operand is. That is the shape of "the highest allele frequency over the ancestry groups that have one":

from altar.predicates import Col, Ge, Greatest, Lit

common = Ge(Greatest(Col("af_afr"), Col("af_nfe"), Col("af_eas")), Lit(0.01))
common.evaluate({"af_afr": None, "af_nfe": 0.02})  # True: one missing group does not hide the others

BigQuery's GREATEST returns NULL when any argument is NULL, while DuckDB's and PostgreSQL's skip NULL arguments. To give the same answer on every engine, and the same as the Python rendering, Greatest renders each argument as a COALESCE of the operands, rotated to start at a different one:

GREATEST(COALESCE(af_afr, af_nfe, af_eas), COALESCE(af_nfe, af_eas, af_afr), COALESCE(af_eas, af_afr, af_nfe))

The SQL text grows with the square of the operand count.

A float NaN is a value, not a null, in both renderings: IsNull is False for it, and Greatest and Least do not skip it. A source emits None (SQL NULL) for a missing value, not NaN.

In values and Lit constants may be NumPy scalars, for example values taken from an array. They render as plain SQL literals, and In stores them as built-in Python scalars with duplicates removed in first-occurrence order. Pass an ordered collection such as a tuple or list; a set is rejected because its order, and so the SQL text, can differ between processes.

Attach a rule to a model or a source

A ModelPlugin returns a Predicate from prioritize_predicate() (Add a model binding), and a prioritizing AnnotationSource returns one as well. The SQLite store runs the Python rendering, and BigQueryScoreStore runs the SQL rendering in its query.

Next