Results and storage¶
altar.results defines score, detail and artifact stores, and assembles stored scores and annotations into
variant results. Store implementations and result readers import from it.
Store and assemble results shows how to use it.
Stable, except a few BigQuery export helpers, the list of a wide table's columns and the long export, which are provisional. Provisional names lists them.
results
¶
Stable score-store, schema, and materialized-result API.
Result-backend authors and consumers should import these names from :mod:altar.results. Backend modules
under altar.plugins.data_lake remain implementation details even when their classes are re-exported here.
The names a host application uses to subclass or configure the BigQuery store are listed in __provisional__:
VariantIdentity, DisplayRider, RiderColumn, export_col_alias and safe_identifier. They are
expected to change. So are the wide export's column manifest (wide_column_manifest, ColumnManifestEntry,
column_manifest_tsv, write_column_manifest, COLUMN_MANIFEST_FILE) and the long export
(write_long_export, long_score_tables, LongScoreTable, LongModelInfo), also provisional.
The result-bucket export is provisional too: the contract a store hands its export over in (ResultBuckets,
ResultBucket, ResultBucketDescriptor, RESULT_BUCKET_SCHEMA_VERSION, result_bucket_schema,
genomic_sort_key, and bucket_export_layout, bucket_rows_for and plan_buckets, with which a store
that produces its own buckets cuts them), the writers that render it into any storage (render_wide_tsv,
render_wide_parquet, render_prioritized_ndjson, render_long_export, render_occurrence_buckets),
BigQuery's staging setting (ExportStaging), the cost guard's refusal (ExportScanLimitExceeded) and the
refusal of an export that repeats a variant (ExportDuplicateRowsError).
LINEAGE_CATALOG_COLUMNS
module-attribute
¶
LINEAGE_CATALOG_COLUMNS: Mapping[str, str] = (
MappingProxyType(
{
"score_lineage_id": "STRING",
"lineage_json": "STRING",
}
)
)
The BigQuery columns and types of a lineage catalog, the table named by lineage_table_id, by column name.
Every column is a scalar and required. The catalog holds one row per registered lineage.
LINEAGE_DESCRIPTOR_COLUMNS
module-attribute
¶
LINEAGE_DESCRIPTOR_COLUMNS: Mapping[str, str] = (
MappingProxyType(
{
"score_lineage_id": "STRING",
"plugin_id": "STRING",
"descriptor_json": "STRING",
"recorded_at": "TIMESTAMP",
}
)
)
The BigQuery columns and types a lineage catalog's descriptor sidecar must have, by column name.
BigQueryScoreStore.lineage_descriptor_table_id names the table. Every column is a scalar, and the table may
carry extra nullable columns.
LINEAGE_PROVENANCE_COLUMNS
module-attribute
¶
LINEAGE_PROVENANCE_COLUMNS: Mapping[str, str] = (
MappingProxyType(
{
"score_lineage_id": "STRING",
"provenance_hash": "STRING",
"run_identity_json": "STRING",
"recorded_at": "TIMESTAMP",
}
)
)
The BigQuery columns and types of a lineage catalog's provenance sidecar, by column name.
BigQueryScoreStore.lineage_provenance_table_id names the table. Every column is a scalar and required. The table
holds one row per exact run identity registered under a lineage.
COLUMN_MANIFEST_FILE
module-attribute
¶
The file the column manifest of a wide export is written to, beside its part.
RESULT_BUCKET_SCHEMA_VERSION
module-attribute
¶
The version of the result-bucket schema this module defines; bumped by any change to its columns.
ArtifactKey
dataclass
¶
ArtifactKey(
job_id: str,
scoring_batch: int | None = None,
shard: tuple[tuple[str, str], ...] = (),
variant_id: str | None = None,
)
The identity of the task that produced one artifact file, and of the variant a per-variant file is for.
job_id and scoring_batch are the request's; shard holds any further identity labels the
plan's task carried (fold, num_folds), as sorted pairs, so a plan whose terminal tasks each emit a
file keeps them apart. variant_id is set only for a variant-scoped artifact: it names the one
variant of the task's batch the file is for, so the files one task writes per variant keep distinct keys.
The key is provenance: it says which task wrote a file, while the record's variant_ids say what it
covers. Records are content-addressed, so a retried batch that produces different bytes under the same key
adds a record rather than replacing one.
from_task_labels
classmethod
¶
from_task_labels(
job_id: str,
scoring_batch: int | None,
labels: Mapping[str, str],
*,
variant_id: str | None = None,
) -> ArtifactKey
Build the key of a container task from the request identity and the task's labels.
The labels every plan sets from the request (job_id, model_id, kind, scoring_batch,
scoring_batch_count) are already in the key or are not identity, so only the remaining labels
become the shard identity. variant_id names the variant of a variant-scoped file.
ArtifactRecord
dataclass
¶
ArtifactRecord(
model_id: str,
artifact_name: str,
lineage_id: str,
key: ArtifactKey,
variant_ids: tuple[str, ...],
media_type: str,
digest: str,
byte_size: int,
uri: str,
)
One stored artifact file: what produced it, what it covers, and where its bytes are.
digest is the sha256: content identity of the bytes and byte_size their length. uri is
the location the store put the bytes at, in the vocabulary of its storage; :meth:ArtifactStore.locate
turns a record into a readable location for a consumer. A record whose key names a variant covers exactly
that variant.
ArtifactStore
¶
Bases: ABC
Persistence axis for binding-declared opaque result files.
Every read takes accepted_lineage_ids, the lineages whose records count, in preference order (the
current run's lineage first, as :meth:~altar.results.ScoreReusePolicy.ordered_ids orders them). The
empty default places no lineage restriction. Writes name their lineage explicitly: the scoring engine
passes the lineage its primary rows will carry.
close
¶
Release the connections and clients this store opened itself.
The default does nothing. A store that opens a connection or creates a client overrides it, and a second
call must do nothing. A storage adapter or client passed to the constructor stays open: whoever created it
closes it. altar.run closes the artifact store it builds from a run spec when the run ends.
ensure_artifact
abstractmethod
async
¶
ensure_artifact(
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
schema: ArtifactSchema,
lineage_id: str | None = None,
) -> None
Create or validate one lineage's artifact without silently changing its schema or plugin identity.
lineage_id is the score lineage of the producing run, the ID its primary rows are stored under; it
defaults to the lineage :meth:~altar.results.ScoreLineage.from_run generates for plugin_identity.
put
abstractmethod
async
¶
put(
model_id: str,
artifact_name: str,
*,
lineage_id: str,
key: ArtifactKey,
variant_ids: Sequence[str],
source_uri: str,
storage: Storage,
digest: str,
) -> ArtifactRecord
Copy the file at source_uri (readable through storage) into the store and index it.
digest is the sha256: identity of the bytes, computed by the caller. The artifact must have
been declared with :meth:ensure_artifact under lineage_id first, and key and variant_ids
must fit its scope (:func:require_artifact_coverage). A put of bytes the lineage already holds under
the same key is idempotent and returns the existing record. The same bytes under another key are
another record that may share the stored bytes, so two variants whose per-variant files happen to be
identical are each covered.
get_missing_variants
abstractmethod
async
¶
get_missing_variants(
model_id: str,
artifact_name: str,
variant_ids: Sequence[str],
accepted_lineage_ids: Sequence[str] = (),
) -> set[str]
Return requested variants no record of this model's artifact lists under an accepted lineage.
read_records
abstractmethod
async
¶
read_records(
model_id: str,
artifact_name: str,
*,
variant_ids: Sequence[str] | None = None,
accepted_lineage_ids: Sequence[str] = (),
) -> AsyncIterator[ArtifactRecord]
Stream stored records, optionally those covering any of variant_ids, under accepted lineages.
locate
abstractmethod
async
¶
locate(record: ArtifactRecord) -> str
Return a location a consumer can read record's bytes from through the store's storage.
describe_artifact
abstractmethod
async
¶
describe_artifact(
model_id: str, artifact_name: str, *, lineage_id: str
) -> StoredArtifact | None
Return durable schema/plugin provenance of one lineage's artifact, or None when it is absent.
StoredArtifact
dataclass
¶
StoredArtifact(
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
schema: ArtifactSchema,
lineage_id: str,
)
Persisted scientific and runtime identity for one model's named artifact under one lineage.
lineage_id is the score lineage of the runs that produced the files: the ID their primary rows carry.
plugin_identity is one representative exact identity of that lineage.
supplies
¶
supplies(schema: ArtifactSchema) -> bool
Whether this artifact's files can stand in for schema when its lineage is accepted for reuse.
Accepting a lineage is the caller's scientific decision about the producer; this is the structural half: the same named artifact, at the same declared version, with the same media type and scope. Descriptions do not matter.
BigQueryScoreStore
¶
BigQueryScoreStore(
client: Any,
table_id: str,
*,
identity: VariantIdentity | None = None,
lineage_table_id: str | None = None,
primary_stage_rows: int = 1000000,
export_staging: ExportStaging | None = None,
export_maximum_bytes_billed: int | None = None,
export_staging_max_jobs: int = DEFAULT_STAGING_MAX_JOBS,
read_client: Any = None,
)
Bases: ScoreStore
BigQuery-backed score store. One table (project.dataset.table) keyed on variant_id and model_id,
holding the locus identity columns (VariantIdentity) and one column per plugin score column (added by
ensure_schema), clustered for locus pruning.
Build a store over the score table table_id, querying it with client.
client is a google.cloud.bigquery.Client or a compatible fake. table_id is
"project.dataset.table". identity is the locus schema, defaulting to VariantIdentity. The client
is injected so this module needs no google-cloud-bigquery at import. primary_stage_rows is the size
of each staging load job a primary_write unit runs (see primary_write); it bounds the rows a unit
buffers in memory and sets how many load jobs an ingest costs against BigQuery's per-table daily quota.
The export settings shape result_buckets (see altar.plugins.data_lake.bigquery_buckets):
export_staging is a GCS location the buckets are staged through with EXPORT DATA, separate from where
the export's files go; without it they are read with the Storage Read API, through read_client, a
google.cloud.bigquery_storage_v1.BigQueryReadClient (or a compatible fake). When it is omitted, the store
asks client for one with bigquery_read_client() if it has that method, or else builds one with client's
credentials, read from its _credentials attribute, which a wrapper or proxy around a real client passes
through. Pass read_client for a client that exposes neither.
export_maximum_bytes_billed refuses an export whose dry-run scan estimate is larger, and caps every
export job's billing; there is no limit by default. export_staging_max_jobs caps the EXPORT DATA jobs
one staged export runs.
client
property
writable
¶
Return the BigQuery client this store runs its queries and load jobs with.
It is read-only. Assigning the client the store already holds is accepted; any other value raises
AttributeError.
table_id
property
writable
¶
Return the score table, as "project.dataset.table".
It is read-only. Assigning the ID the store already holds is accepted; any other value raises
AttributeError.
lineage_provenance_table_id
property
¶
Return the lineage catalog's provenance sidecar, "<lineage_table_id>__provenance", or None.
It is None when the store was built without lineage_table_id.
lineage_descriptor_table_id
property
¶
Return the lineage catalog's descriptor sidecar, "<lineage_table_id>__descriptors", or None.
It is None when the store was built without lineage_table_id. LINEAGE_DESCRIPTOR_COLUMNS lists
the columns the table must have. The store never creates it except through ensure_descriptor_table.
connect
classmethod
¶
connect(
table_id: str,
*,
credentials: Any = None,
lineage_table_id: str | None = None,
identity: VariantIdentity | None = None,
primary_stage_rows: int = 1000000,
export_staging: ExportStaging | None = None,
export_maximum_bytes_billed: int | None = None,
export_staging_max_jobs: int = DEFAULT_STAGING_MAX_JOBS,
) -> BigQueryScoreStore
Build a store against a real BigQuery client. Requires the altar[bigquery] extra.
The store owns the client it builds here, and close releases it. The other arguments are the
constructor's.
from_options
classmethod
¶
from_options(
*,
table_id: str,
connection: Connection,
lineage_table_id: str | None = None,
primary_stage_rows: int = 1000000,
export_staging_uri: str | None = None,
export_maximum_bytes_billed: int | None = None,
export_staging_max_jobs: int = DEFAULT_STAGING_MAX_JOBS,
) -> BigQueryScoreStore
Build a store from a run spec's [store] options, with a client from connection.bigquery_client().
table_id, lineage_table_id, primary_stage_rows, export_maximum_bytes_billed and
export_staging_max_jobs are the constructor's. export_staging_uri, a gs:// prefix, stages exports
there, read back through an FsspecStorage on the same connection. The run driver passes the section's
named connection; a spec names the connection and never carries a client or credential. The store owns
the client it builds here, and close releases it. The connection only says how to reach BigQuery: it
never enters a score's identity, so reading the same table through another connection reuses every stored
score.
close
¶
Close the BigQuery clients connect, from_options or an export built; a second call does nothing.
A client passed to the constructor is left open.
result_buckets
¶
result_buckets(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
rider: DisplayRider | None = None,
bucket_rows: int | None = None,
bucket_bytes: int | None = None,
variants_relation: str | None = None,
maximum_bytes_billed: int | None = None,
) -> AbstractAsyncContextManager[ResultBuckets]
Produce the export as result buckets from one canonical table (see ScoreStore.result_buckets).
One CREATE TABLE … AS reads the score table and the annotation sources once into a per-variant table in
the score table's dataset, and the buckets are read from it, through export_staging when it is set and
the Storage Read API otherwise (altar.plugins.data_lake.bigquery_buckets). The table and any staged
shards are removed when the block exits. variants_relation scopes a warehouse-scale job as in
materialize; more than 50,000 inline variant_ids are loaded into a scratch table. A dry run estimates
the scan first: over maximum_bytes_billed (default: the store's export_maximum_bytes_billed) it raises
ExportScanLimitExceeded before any export job runs. Every annotation source must join in SQL. Without models,
the portable path annotates the variants.
ensure_schema
async
¶
ensure_schema(columns: list[ScoreColumn]) -> None
Create the score table, or add the score columns and the identity column it lacks.
The table's metadata is read first. A column whose physical type differs from the requested one raises
SchemaCollisionError. When the table already has every column, no DDL statement runs. BigQuery counts
each DDL statement toward the table's daily operation limit, even one that changes nothing. A client
that cannot look up table metadata always runs the CREATE TABLE IF NOT EXISTS and ADD COLUMN IF NOT
EXISTS statements. With a lineage catalog, the catalog and its provenance sidecar are created the same
way, only when missing.
add_scores_frame
¶
add_scores_frame(
frame: Any,
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
columns: list[ScoreColumn],
lineage_descriptor: LineageDescriptor | None = None,
) -> InsertResult
Ingest scores from a pandas DataFrame the caller already holds.
This is the columnar path for a caller that already has the scores as a frame — for example a scoring
pipeline reading its result TSV in pd.read_csv chunks. It skips add_scores' round-trip of rows to
per-row dicts to from_records. frame carries the decomposed locus columns (VariantIdentity:
chr/pos/ref/alt) plus one column per score in columns. variant_id is derived here from the
locus (chr:pos:ref:alt), and genome (the run's genome_build), model_id, and model_name are
stamped on, so the caller supplies only loci and scores. The locus must already be canonical (chr1,
not 1; upper-case alleles); a non-canonical one raises VariantIdentityError before anything loads.
A missing score is NaN or None in the frame and loads as NULL; an infinite score raises ValueError.
This method is synchronous and blocking, because the columnar bulk load is a blocking BigQuery job, so
it is not part of the async ScoreStore interface. A caller on an event loop should offload it to a
worker thread (e.g. asyncio.to_thread) so a multi-minute load doesn't stall the loop. The
rows-based add_scores is the interface contract; this is a columnar fast path warehouse-scale callers
can opt into. With compact lineages, a lineage_descriptor is verified and recorded before the lineage
is registered.
stage_scores_frame
¶
stage_scores_frame(
frame: Any,
*,
staging_table: str,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
columns: list[ScoreColumn],
replace: bool = False,
) -> int
Load one result chunk into a job-specific staging table, not the canonical score table.
The first chunk uses replace=True; later chunks append. The staging table is given a seven-day
expiry immediately after its first load, so an interrupted completion cannot leak scratch tables.
Call :meth:promote_staged_scores after the complete file is staged, then
:meth:drop_staged_scores in a finally block. :meth:primary_write composes exactly that
lifecycle over portable rows with a store-chosen staging table, and is what the scoring engine and
:func:~altar.scoring.ingest_scoring_results use.
promote_staged_scores
async
¶
promote_staged_scores(
staging_table: str,
*,
model_id: str,
plugin_identity: PluginRunIdentity,
columns: list[ScoreColumn],
lineage_descriptor: LineageDescriptor | None = None,
) -> InsertResult
Append a complete, identity-checked staged result to the canonical table.
BigQuery does not enforce unique keys, so concurrent jobs may append the same score key. Read paths
defensively select one physical row per key, and deployments may compact old duplicates periodically.
The bounded staging source is deduplicated here as an additional guard against repeated scorer rows: each
key keeps one whole staged row, the newest by created_at, with its values as a deterministic tie-break.
Promotion independently checks that the staging table contains exactly the expected model and plugin run identity, then refreshes the canonical identity for that model. The insert is also filtered to the expected pair, so rows appended to the staging table between validation and insertion cannot cross the identity boundary.
Deliberately do not MERGE score keys against the canonical cache: it is an unpartitioned,
multi-billion-row table, and a source-correlated join cannot prune its clustered blocks. The one
canonical read is clustered by model_id and exists only to enforce run identity; score-key
promotion still scales with the staged batch rather than the full cache.
With compact lineages, a lineage_descriptor (verified against plugin_identity first) is recorded in
the descriptor sidecar before the lineage is registered; the sidecar must be provisioned.
drop_staged_scores
async
¶
Best-effort caller-facing primitive for removing a score staging table.
primary_write
¶
primary_write(
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
columns: list[ScoreColumn],
lineage_descriptor: LineageDescriptor | None = None,
) -> AbstractAsyncContextManager[PrimaryScoreWriter]
Stage every batch of the unit in a private staging table, promote it on commit, and always drop it.
This composes stage_scores_frame's load, promote_staged_scores, and drop_staged_scores, so a host
no longer calls them itself. The staging table is chosen here, in the score table's dataset, named after
the score table, the model, and a random suffix, so concurrent units never share one; the caller never
sees it. Each add assembles its rows as add_scores does into a columnar buffer and returns the rows
it accepted as added. The buffer is loaded into the staging table in load jobs of exactly
primary_stage_rows rows (a constructor option) as it fills, so an ingest of N rows runs
ceil(N / primary_stage_rows) load jobs rather than one per engine batch, and the buffer never holds a
full load's worth of rows between adds; the first load replaces the table and sets its seven-day
expiry. A clean exit loads what remains in the buffer, then promotes the staged rows, which validates
the staged model and run identity, rechecks the canonical table's run identity, and (with compact
lineages) registers the lineage and the run's provenance, as add_scores does; a unit that accepted no
rows loads and promotes nothing. An exception aborts the unit: the buffer is discarded unloaded and
nothing is promoted, so none of its rows and none of its lineage provenance reach the canonical tables.
Either way the staging table, if a load reached it, is dropped; a failed drop is logged rather than
raised (it would otherwise mask the unit's own error), and the expiry removes the table later.
The unit runs ensure_schema(columns) when it opens unless this store already ensured those columns,
since promotion inserts into them by name. Without compact lineages the canonical run identity is
checked when the unit opens, so an incompatible
run fails before anything is staged, and again at promotion. The promotion is one INSERT … SELECT
statement, so a commit is all-or-nothing in BigQuery too; like every write to this append-only table it
does not dedup against existing keys.
A lineage_descriptor is verified when the unit opens and, with compact lineages, the provisioned
descriptor sidecar is checked then too, so a missing sidecar fails before anything is staged; promotion
records the descriptor.
get_unscored_variants
async
¶
get_unscored_variants(
variant_ids: Sequence[str],
model_id: str,
*,
variants_relation: str | None = None,
chromosomes: Sequence[str] | None = None,
) -> list[str]
Return the variant_ids not yet scored by model_id, so a scoring pipeline can skip cache hits.
By default the ids are bound inline as @variant_ids. Pass variants_relation (a backtick-quoted
temp-table ref carrying variant_id) to prune at warehouse scale as materialize does, for a variant
set too large to bind as an array param. On the relation path the full input lives only in the temp
table, so the unscored rows are read straight off a LEFT JOIN … IS NULL dedup and variant_ids may be
empty, rather than diffed against an in-memory list. chromosomes then prunes the scores table's chr
cluster key, as in export_unscored; it is ignored on the inline path, whose small array binds
directly.
Both paths filter on the model's genome build, read from its stored run identity. This store keeps one
scientifically compatible identity per model_id, and that identity includes the genome build, so the
filter only prunes the genome cluster key and never hides a compatible score. A model with no stored
identity has no scores, and the query then runs without a genome filter.
get_missing_scores
async
¶
get_missing_scores(
variant_ids: Sequence[str],
model_id: str,
*,
plugin_identity: PluginRunIdentity,
columns: Sequence[ScoreColumn],
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
variants_relation: str | None = None,
chromosomes: Sequence[str] | None = None,
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
) -> list[str]
Filter only candidates, using explicit input genome and the resolved lineages.
variants_relation is the warehouse-scale equivalent of the inline ID
list. For large miss sets, use export_unscored to keep them in BigQuery.
lineage_descriptor and equivalence resolve as in resolve_lineages.
prepare_unscored_table
async
¶
prepare_unscored_table(
*,
dest_table: str,
variants_relation: str,
model_id: str,
plugin_identity: PluginRunIdentity,
columns: Sequence[ScoreColumn],
max_variants_per_batch: int,
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
chromosomes: Sequence[str] | None = None,
expiration_timestamp: str = "TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)",
variant_eligibility: VariantEligibility | None = None,
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
detail_store: BigQueryDetailStore | None = None,
detail_schemas: Sequence[DetailSchema] = (),
artifact_schemas: Sequence[ArtifactSchema] = (),
check_representation: bool = False,
reference_fasta: str | PathLike[str] | None = None,
ineligible_table: str | None = None,
) -> int
Write every cache miss of the candidates in variants_relation to dest_table, numbered into batches.
This is the warehouse form of prepare_scoring_batches, and it sets aside and counts as a miss the same
variants. Returns the number of misses written. dest_table holds the locus columns, variant_id and
a zero-based scoring_batch, and expires at expiration_timestamp. Export each batch with
export_prepared_batch, which reads only this table. The caller owns the candidate relation, the
destination and the scoring requests. The candidate relation holds the loci and their variant_id, for
the request's genome build.
A candidate is a miss unless an accepted lineage holds its primary row and, for each table in
detail_schemas, detail_store holds at least one row for it under a lineage whose table supplies that
schema. A detail table whose row_count_field is 0 in the cached primary row needs no row. Pass the
plugin's manifest.detail_schemas and the BigQueryDetailStore scoring writes to. The accepted lineages
are resolved as in resolve_lineages, with lineage_descriptor and equivalence; a host that froze a
resolution passes ScoreReusePolicy.exactly(...) instead. This query cannot check artifact coverage, so
a request whose output selection names artifacts (artifact_schemas, from the manifest's
selected_artifact_schemas) raises CapabilityNotSupported.
These candidates are set aside rather than prepared, each with the first reason that applies:
- a
variant_idthat is not canonical, or does not spell the locus in the other columns, isnot_normalized; variant_eligibility, the model'smanifest.variant_eligibility, excludes a candidate asunsupported_variant_classorallele_too_long;- with
check_representation, alleles that are not in minimal representation arenot_normalized.
By default no representation check runs, so a canonical candidate whose alleles are not minimal is
prepared, as before these checks existed. Pass check_representation=True to set it aside, as
prepare_scoring_batches does. These checks run in SQL, so no locus is read to the host. The misses are
written to a scratch table in the score table's dataset, and dest_table numbers them into batches from
there.
reference_fasta, an indexed FASTA of the run's genome build, turns the representation check on and checks
it against the reference. The misses are read back and checked as preparation checks them
(representation_check: contig_missing, reference_mismatch, then not_normalized against left
alignment), and the rejected loci are left out of dest_table. That read holds at most one result page,
10,000 loci and 100,000 rejected loci in memory at a time.
Pass ineligible_table, written like dest_table, to keep the set-aside candidates. It is replaced with
one row per candidate in the columns of preparation's report (INELIGIBLE_VARIANT_COLUMNS), and expires
at expiration_timestamp. Without it, the counts are logged as a warning.
count_unscored
async
¶
count_unscored(
*,
variants_relation: str,
model_id: str,
plugin_identity: PluginRunIdentity,
columns: Sequence[ScoreColumn],
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
chromosomes: Sequence[str] | None = None,
variant_eligibility: VariantEligibility | None = None,
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
detail_store: BigQueryDetailStore | None = None,
detail_schemas: Sequence[DetailSchema] = (),
artifact_schemas: Sequence[ArtifactSchema] = (),
check_representation: bool = False,
) -> int
Count the cache misses prepare_unscored_table would find, in one query that writes nothing.
The arguments mean what they mean for prepare_unscored_table, and the count is the one it returns without
a reference_fasta. The reference check reads every miss on the host, so this count is taken before it, and
a preparation with a reference can set aside more. The lineage catalog is read and never written. This
method is provisional.
export_prepared_batch
async
¶
export_prepared_batch(
*,
prepared_relation: str,
scoring_batch: int,
destination_uri: str,
) -> int
Export one previously prepared miss batch without rescanning canonical scores.
The prepared table holds only the loci prepare_unscored_table kept after its eligibility checks.
export_unscored
async
¶
export_unscored(
*,
model_id: str,
destination_uri: str,
variants_relation: str,
chromosomes: Sequence[str] | None = None,
scoring_batch: int | None = None,
scoring_batch_count: int | None = None,
score_cache_cutoff: datetime | None = None,
scoring_batch_size: int | None = None,
plugin_identity: PluginRunIdentity | None = None,
columns: Sequence[ScoreColumn] = (),
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
variant_eligibility: VariantEligibility | None = None,
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
) -> int
Export the loci of variants in variants_relation not yet scored by model_id, and return the count
exported. Writes with EXPORT DATA to destination_uri, an already-resolved gs://…-*.tsv wildcard
prefix, staging each model's unscored-variant file. This is a sibling of export_download and
export_prioritized; it differs only in the predicate (LEFT JOIN scores … WHERE s.variant_id IS NULL)
and a flat headerless locus projection.
variants_relation (a backtick-quoted temp-table ref) is required, and unlike materialize's relation
it must carry the decomposed locus columns (VariantIdentity: chr/pos/ref/alt), because the
export projects them; the caller's temp table must produce exactly those columns. chromosomes prunes
the scores table's chr cluster key; omit it for a correct but unpruned scan. The genome filter is
plugin_identity.genome_build when a run identity is given, otherwise the model's stored genome build.
First a parameterized COUNT(*) runs, returning 0 and skipping the export when nothing is unscored.
Then the EXPORT DATA OPTIONS(… CSV, tab, header=false) AS dedup SELECT runs. EXPORT DATA forbids
query params, so genome, model_id, and chromosomes are inlined into the export SQL (they are
controlled internal values) while the count binds them as params.
Pass the model's manifest.variant_eligibility as variant_eligibility so the candidate relation is
restricted to variants the model can score before any miss is counted, batched, or exported. Without it an
SNV-only model's export includes indels, and run_scoring then rejects their result rows. The manifest's
rules are all this export checks; export_unscored_checked also applies preparation's reference and
normalization checks to the exported loci and reports the ones it sets aside.
Resolving the gs:// URI, creating the temp table, and composing the resulting shards stay the caller's
job; the store's job ends at issuing the export. This does not run on the emulator: the EXPORT DATA GCS
sink hangs the job poll, as in export_download, so the generated SQL is checked with a fake client for
shape plus an opt-in smoke test against real BigQuery. In compact-lineage mode, lineage_descriptor and
equivalence resolve the accepted lineages as in resolve_lineages.
export_unscored_checked
async
¶
export_unscored_checked(
*,
storage: Storage,
ineligible_report_uri: str,
reference_fasta: str | PathLike[str] | None = None,
model_id: str,
destination_uri: str,
variants_relation: str,
chromosomes: Sequence[str] | None = None,
scoring_batch: int | None = None,
scoring_batch_count: int | None = None,
score_cache_cutoff: datetime | None = None,
scoring_batch_size: int | None = None,
plugin_identity: PluginRunIdentity | None = None,
columns: Sequence[ScoreColumn] = (),
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
variant_eligibility: VariantEligibility | None = None,
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
) -> UnscoredExport
Run export_unscored, then set aside the exported loci the reference rejects or that are not normalized.
This is the host-side check prepare_scoring_batches applies, for the relation-native export: without
it, one locus with an unknown contig or a REF that disagrees with the FASTA fails a whole scoring batch
after it was dispatched. The export arguments mean what they mean for export_unscored (pass the
manifest's variant_eligibility there, as before; it filters in SQL). Each exported file is then read
back through storage and checked with the same rules and reasons as preparation
(altar.plugins.model.preparation.representation_check): with reference_fasta, the indexed FASTA of
the run's genome build, contig_missing, reference_mismatch, then not_normalized against left
alignment; without one, only not_normalized against minimal representation. A file that held an
ineligible locus is rewritten in place with only its eligible loci, so the batch files a host
dispatches hold exactly the eligible loci. The removed loci are written to ineligible_report_uri as
the headered TSV preparation writes (INELIGIBLE_VARIANT_COLUMNS, in file order); when none are
removed, a report left there by an earlier run is deleted, so the report always describes this export.
destination_uri must hold exactly one *. BigQuery numbers an export's files sequentially from
000000000000, so the files are found by probing those names until one is missing (Storage has no
listing); they are returned as shard_uris. ineligible_report_uri must not match the wildcard, or a
later listing would read it as a batch file. Reading back more loci than the export counted means files
of an earlier, larger export remain under the prefix, and raises RuntimeError; clear the prefix first.
The returned exported counts the loci actually read back, which a concurrent writer can make smaller
than export_unscored's count.
materialize
async
¶
materialize(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
variants_relation: str | None = None,
) -> AsyncIterator[MaterializedVariant]
Stream one per-variant summary at a time from a single fused query.
The query is one SELECT that LEFT JOINs the scores table to each annotation source's sql_join
JoinSpec. It computes each model's prioritize_predicate().to_sql() server-side as a per-row
is_prioritized column — the same generated SQL the EXPORT paths filter on — and each prioritizing
source's predicate as a boolean column. SQL decides prioritization; the model_scores entries are then
shaped by each model's to_model_score in Python. A parity test checks the result matches the pure
materialize_variant driver row for row, so a caller can trust the pushed-down query.
Streaming: rows arrive ordered by variant_id, so grouping yields one MaterializedVariant at a time
without buffering the whole result, which can span millions of variants. That order is the
ScoreStore.materialize contract: one row per distinct requested variant, ascending by variant_id.
Scope: one MaterializedVariant per distinct requested variant, as with the portable driver. The fused
query covers every variant in variants_relation when one is given. On the inline path it covers the
variants a model scored plus those a fused prioritizing source has a row for (see build_scored_cte), so
a coding-only variant (e.g. an AlphaMissense missense row with no ChromBPNet score) comes back with empty
model_scores and prioritized driven by the source. The remaining requested variants — ones with only
passive annotations, or with no evidence at all — are built in Python with empty model_scores, their
annotations read and prioritizing sources evaluated over fetch_annotations, and merged into the sorted
stream in variant_id order. Every Python annotation read (these, and the annotate() fallback below)
fetches a chunk of _MATERIALIZE_CHUNK_VARIANTS sorted requested ids at a time as the sorted stream
reaches them, so only one chunk's annotations per read are held.
Sources on this backend join in-SQL (sql_join returns a spec); a source on another backend falls back to
annotate(), plus Python evaluation when it prioritizes. A passive column referenced by a model
predicate must come from a fused source, since it has to be in the query for the predicate to push down.
Annotations: each result's annotations carries the fused sources' columns, which the query already
selects for the predicates, plus the other sources' annotate() values, with the same values and the
same absent-when-empty rule as the portable driver. A declared scalar json column that the query returns
as JSON text (a STRING such as TO_JSON_STRING(...)) is decoded to its JSON value first. The unscored
variants outside the fused query read only the fused passive sources again; a fused prioritizing source
has no row for them, and the other sources' values were already read. A relation-scoped call with a
source that cannot fuse raises ValueError unless the job's ids are also passed in variant_ids, since
annotate() needs the ids.
Scoping: variant_ids are bound inline as @variant_ids. Pass variants_relation (a temp-table ref)
instead to scope a warehouse-scale job whose id array would exceed BigQuery's request limit; the SQL
then prunes via variant_id IN (SELECT variant_id FROM <relation>) and variant_ids may be empty. The
two paths return the same result over the same variant set (checked on the emulator). More than 50,000
distinct variant_ids without a variants_relation are loaded into a scratch table in the score table's
dataset with a load job, and the query is scoped by that table. The table is dropped once the query has
run, and it expires after a day if the drop fails.
export_download
async
¶
export_download(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
destination_uri: str,
rider: DisplayRider | None = None,
variants_relation: str | None = None,
prioritized_only: bool = False,
header: bool = True,
scored_relation: str | None = None,
anno_relation: str | None = None,
storage: Storage | None = None,
) -> str
Export variant_ids as a flat wide TSV to destination_uri (a gs:// prefix) and return it. This is
the download branch of the two exports. prioritized_only restricts to prioritized variants (the same
rollup as export_prioritized, but in the wide-TSV shape) for the prioritized download. header=False
writes headerless parts so a caller doing a server-side GCS compose can prepend a single header (see
download_header) instead of one per shard. storage is the base signature's writer; the bulk
EXPORT DATA job writes to the prefix itself, so it is accepted and unused.
This is a different artifact from materialize: all variants (not prioritized-only), one row per
variant, with each model's score columns pivoted into positional columns (MAX(IF(model_id=…))) rather
than a nested ARRAY<STRUCT>. It is the human-downloadable TSV, not the DB rows. It covers the same
universe as materialize, every requested variant whether or not a model scored it (inline, the bound
@variant_ids; see requested_universe_arm). It shares the same
join and pushed-down is_prioritized base (the scored CTE) with materialize, then aggregates:
ANY_VALUE per annotation column, and prioritized = COALESCE(LOGICAL_OR(is_prioritized), FALSE) OR …
over the fused sources. It runs as one EXPORT DATA OPTIONS(... CSV, tab, GZIP) job. BigQuery cannot
export nested or repeated data as CSV, so each annotation column declared repeated or struct is
written as its TO_JSON_STRING text, with a missing value left empty. Such a column must physically be
an ARRAY or STRUCT, not a pre-serialized string. A scalar json column is exported unchanged. The Parquet
and JSON exports keep those columns' native types.
Fused sources only: a flat cloud EXPORT is single-backend, so unlike materialize there is no
annotate() fallback. A source that cannot sql_join raises ValueError.
rider is an optional config-injected DisplayRider — for example most_active_celltype and
most_active_celltype_logfc display columns, the single model-score row picked per variant by an
ordering over the scored namespace. It applies to this aggregated export and the DB export, never to
materialize.
This does not run on the emulator: EXPORT DATA … OPTIONS(uri='gs://…') hangs the job poll on the
emulator's GCS sink, so the generated SQL is checked with a fake client for shape and the actual GCS
write is an opt-in smoke test against real BigQuery. variants_relation scopes as in materialize (a
temp-table ref for a warehouse-scale job; None for inline @variant_ids).
scored_relation reads a pre-materialized scored table (from materialize_scored_table) instead of
rebuilding and rescanning the fused base CTE. When set, the join and pushed-down is_prioritized are not
recomputed here; the export reads the already-computed table, so several exports off one base pay the
base scan (which can be multi-TB) once instead of once each. variants_relation and variant_ids are
then unused, since the table already carries the job's scope.
anno_relation is the small per-variant table from materialize_annotations_table, paired with a
scores-only scored_relation from materialize_scores_table. The export reconstructs scored by
LEFT-JOINing them, so each export scans the small scores-only and annotations-once tables instead of a
base that duplicates every variant's annotation blobs across all its model rows. This cuts the residual
cost below the single-table scan-once path.
export_occurrence_buckets
async
¶
export_occurrence_buckets(
*,
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
occurrence_relation: str,
result_relation: str,
destination_uri: str,
max_record_ordinal: int,
bucket_size: int,
rider: DisplayRider | None = None,
scored_relation: str | None = None,
anno_relation: str | None = None,
variant_ids: Sequence[str] = (),
variants_relation: str | None = None,
) -> list[str]
LEFT JOIN canonical results to occurrences and export one bounded Parquet prefix per ordinal bucket.
The results are the wide download's rows (build_export_query) materialized into result_relation
for the job's variants, so each bucket row is the occurrence columns, result_variant_id, and the
wide layout's columns; altar.plugins.data_lake.export.write_occurrence_buckets writes the same shape
from any store. The job scope is variants_relation (a temp table, typically the ingest's canonical
ids) or the inline variant_ids, bound as query parameters of the CTAS exactly as export_download
binds them; with a pre-materialized scored_relation the table already carries the scope and neither
is needed. Without any of the three the CTAS would reference unbound parameters, so it is refused
before any query runs.
Relations and the destination are trusted operator-generated identifiers. Bucket numbers and size are validated integers. The method intentionally issues one EXPORT per bucket: BigQuery object names do not encode row ordering, while the bucket boundary gives the streaming writer a bounded sort unit.
download_header
¶
download_header(
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
rider: DisplayRider | None = None,
) -> list[str]
Return the ordered column names of the export_download wide TSV.
This is the single header line to prepend when the export is written headerless (header=False) for a
server-side GCS compose. The columns are the shared wide layout
(altar.plugins.data_lake.export.wide_layout, the one build_export_query renders its SELECT aliases from;
a test locks this): variant_id, prioritized, the rider's display columns, each fused annotation column,
one block per model named with a readable model-name prefix, stable model-id digest, and field suffix, then
prioritized_model_ids and prioritized_source_names.
export_prioritized
async
¶
export_prioritized(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
destination_uri: str,
rider: DisplayRider | None = None,
variants_relation: str | None = None,
scored_relation: str | None = None,
anno_relation: str | None = None,
storage: Storage | None = None,
) -> str
Export only the prioritized variants as NDJSON to destination_uri (a gs:// prefix) and return
it. This is the database branch of the two exports. Each row emits model_scores as a native
ARRAY<STRUCT>, so a loader can copy a shard into a database and decode each row with one json.loads.
That database load is the caller's step, not this method. storage is the base signature's writer;
the bulk EXPORT DATA job writes to the prefix itself, so it is accepted and unused.
It differs from export_download in two ways. It keeps only prioritized variants, via a server-side
HAVING on the same prioritized rollup over the same requested universe (inline, the bound
@variant_ids). It emits model_scores as a native ARRAY<STRUCT>
(ARRAY_AGG(STRUCT(…))) rather than pivoting the scores into positional columns, so the exported JSON
already has the model_scores shape the database column expects. It shares the fused scored CTE (the
join plus the pushed-down is_prioritized) with the other two export surfaces, then aggregates per
variant into the full MaterializedVariant contract:
model_scores— one struct per model that scored the variant, into_model_scorefield order (identity, score columns,COALESCE(is_prioritized, FALSE)). A coding-only variant's single model-less row is dropped (IF(model_id IS NULL, NULL, …) IGNORE NULLS), leaving an empty array.prioritized_model_ids— models whoseis_prioritizedisTRUE. The three-valuedNoneandFALSEare excluded.prioritized_source_names— each fused prioritizing source'snamewhen its flag fired, in declaration order.
Fused sources only: a cloud EXPORT runs on one backend, so unlike materialize there is no
annotate() fallback. A source that cannot sql_join raises ValueError. rider is the same optional display rider as export_download, applied to this
aggregated export and never to materialize.
This does not run on the emulator: the EXPORT DATA … OPTIONS(uri='gs://…', format='JSON') GCS sink
hangs the job poll, so the generated SQL is checked with a fake client for shape and the actual write is
an opt-in smoke test against real BigQuery. The inner SELECT (the ARRAY_AGG(STRUCT) aggregation plus
the HAVING filter) does run on the emulator and is parity-tested via build_prioritized_query.
variants_relation scopes as in materialize: a temp-table ref at warehouse scale, or None for inline
@variant_ids. scored_relation reads a pre-materialized scored table instead of rebuilding the base
CTE; pairing it with anno_relation reads the scores-only plus annotations-once split (see
export_download).
materialize_scored_table
async
¶
materialize_scored_table(
*,
dest_table: str,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
variants_relation: str | None = None,
expiration_timestamp: str | None = None,
) -> str
Materialize the fused scored base once into dest_table with a CTAS, and return dest_table. The
join, the pushed-down is_prioritized, and the per-source flags are computed in a single
CREATE OR REPLACE TABLE … AS pass over the base scores table, which can be multi-TB. The two downloads
and the prioritized NDJSON then export off this table via scored_relation, each a small job-scoped
scan, instead of rebuilding and rescanning the base CTE. A job that emits N surfaces scans the base once
rather than N times.
The written table is exactly SELECT * FROM scored: one row per (variant, model) carrying variant_id,
model_id, model_name, the union of score_columns, every fused annotation column, is_prioritized,
and one boolean per prioritizing source. That is the full column set the scored_relation exports
project on. variants_relation and variant_ids scope the base scan as in materialize, and the
table's universe is that complete scope: a requested variant no model scored gets one model-less row,
as materialize yields it, whichever way the scope is given.
expiration_timestamp is a SQL TIMESTAMP expression (e.g.
TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 6 HOUR)) that sets OPTIONS(expiration_timestamp=…) so a
crashed run's temp table self-expires; None writes no expiration and the caller drops the table.
There is no ORDER BY here, unlike materialize's streaming read, so the CTAS streams rows to the table
without the global sort that pushed the prioritized export past its memory limit. This does not run on
the emulator (a CTAS over the GCS-backed base is real-BigQuery only, like EXPORT); the generated SQL is
checked with a fake client for shape, and the base CTE it wraps is the same emulator-tested
build_scored_cte.
materialize_scores_table
async
¶
materialize_scores_table(
*,
dest_table: str,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
variants_relation: str | None = None,
expiration_timestamp: str | None = None,
) -> str
Materialize the scores-only base into dest_table with a CTAS, and return dest_table. This is the
scores half of the annotations-once split. It writes one row per (variant, model) carrying only
variant_id, model_id, model_name, and the union of score_columns — no annotation columns and no
is_prioritized. The is_prioritized predicate reads region_type, a per-variant annotation, so it
can't be computed here and is deferred to the export join. This is a straight projection of the base
scores table over the job scope, with no annotation joins and no universe UNION, so the written table
holds just the numeric score rows even though the multi-TB base is scanned once.
Pair it with materialize_annotations_table, whose per-variant rows span the same job scope, and export
via scored_relation=<this> and anno_relation=<that>.
variants_relation and variant_ids scope the scan as in materialize. This does not run on the
emulator (a CTAS over the GCS-backed base is real-BigQuery only); the generated SQL is checked with a
fake client for shape.
materialize_annotations_table
async
¶
materialize_annotations_table(
*,
dest_table: str,
annotation_sources: Sequence[AnnotationSource],
variant_ids: Sequence[str],
variants_relation: str | None = None,
expiration_timestamp: str | None = None,
) -> str
Materialize the per-variant annotations-once table into dest_table with a CTAS, and return
dest_table. This is the annotations half of the split. It writes one row per universe variant carrying
every fused annotation column — the JSON blobs stored once, not duplicated across a variant's model rows
— plus one boolean per prioritizing source (see build_annotations_cte). The universe is the complete
job scope, variants_relation or the inline variant_ids, so a requested variant with no result keeps
its row and this pass reads only each annotation source once. The base scores table is not rescanned.
Pair it with materialize_scores_table and export via scored_relation and anno_relation. This
method sees no models, so pass only the sources applicable_annotation_sources keeps for them.
variants_relation and variant_ids also prune each fused source's {variant_filter} as in
materialize. This does not run on the emulator (a CTAS is real-BigQuery only); the generated SQL is
checked with a fake client for shape, and the two-table export join it feeds is emulator-tested via
build_export_query and build_prioritized_query with both relations set.
UnscoredExport
dataclass
¶
UnscoredExport(
exported: int,
eligible: int,
ineligible: int,
ineligible_report_uri: str | None,
shard_uris: tuple[str, ...],
)
What BigQueryScoreStore.export_unscored_checked exported for one scoring batch, and what it set aside.
exported
instance-attribute
¶
Loci the export wrote: every row read back from its files (0 when nothing was unscored).
eligible
instance-attribute
¶
Loci left in the batch files after the ineligible ones were removed.
ineligible
instance-attribute
¶
Loci removed because the reference rejects them or they are not normalized.
ineligible_report_uri
instance-attribute
¶
The ineligible-variant report, or None when nothing was set aside (a stale report is then deleted).
shard_uris
instance-attribute
¶
The export's files in shard order, each holding only eligible loci; some may be empty.
VariantIdentity
dataclass
¶
VariantIdentity(
locus_columns: tuple[tuple[str, str], ...] = (
("chr", "str"),
("pos", "int"),
("ref", "str"),
("alt", "str"),
),
clustering: tuple[str, ...] = (
"genome",
"model_id",
"chr",
"variant_id",
),
)
The locus columns a BigQuery score table carries in addition to the variant_id join key.
These are genome plus the decomposed chr/pos/ref/alt locus, plus the clustering keys, so the
store can cluster and prune scans by locus. variant_id is the portable join key the read and materialize
code is built on; these columns are the source-of-truth locus and cluster keys the caller populates, and
variant_id is chr:pos:ref:alt derived from them. A deployment can override the locus columns or the
clustering. There is no default genome: genome is always the scoring run's genome_build.
ExportStaging
dataclass
¶
ExportStaging(uri: str, storage: Storage)
Where BigQueryScoreStore stages an export's Parquet shards, and how it reads them back (provisional).
uri is a gs:// prefix (BigQuery's EXPORT DATA writes only to GCS) and storage a Storage that reads
and deletes objects under it, such as an FsspecStorage. It is a store setting, separate from the Storage
the export's files are written to, which can be anything.
BigQueryDetailStore
¶
Bases: DetailStore
Generic BigQuery adapter for every binding-declared :class:DetailSchema.
table_id names the canonical JSON-row table. Its metadata sidecar is named
{table_id}__tables and exact compatible provenance is retained in {table_id}__provenance. All
schemas are fixed and contain no binding- or model-specific columns.
Writes stage one bounded batch and MERGE on (model_id, detail_name, lineage_id, logical_key),
making retries idempotent while preserving null-bearing logical keys through their canonical digest.
connect
classmethod
¶
connect(
table_id: str, *, credentials: Any = None
) -> BigQueryDetailStore
Build a store against a real BigQuery client. Requires the altar[bigquery] extra.
The store owns the client it builds here, and close releases it.
from_options
classmethod
¶
from_options(
*, table_id: str, connection: Connection
) -> BigQueryDetailStore
Build a store from a run spec's [detail_store] options, with connection.bigquery_client().
The store owns the client it builds here, and close releases it. The connection never enters any
identity, and a spec never carries the client or a credential.
close
¶
Close the BigQuery client connect or from_options built; a second call does nothing.
A client passed to the constructor is left open.
get_missing_variants
async
¶
get_missing_variants(
*,
model_id: str,
detail_name: str,
variant_ids: Sequence[str],
accepted_lineage_ids: Sequence[str] = (),
) -> set[str]
Return requested variants without rows, reading only the clustered identity columns.
The rows table is unpartitioned and CLUSTER BY model_id, detail_name, lineage_id, variant_id. Each
chunk filters by equality on the first two clustering columns and IN UNNEST on the others, so
BigQuery can prune storage blocks to the requested model/table, lineages, and variants instead of
scanning the table. Only those narrow STRING columns are referenced; row_json is never read, so a
wide detail table (thousands of Enformer tracks per variant) adds rows but not payload bytes to the
scan. One query runs per 10,000 IDs; :func:~altar.scoring.prepare_scoring_batches already bounds
each call by its cache_query_size (default 10,000) and passes only primary cache hits.
DetailRowKey
dataclass
¶
Backend-neutral logical identity for one repeated result row.
digest
property
¶
Return the canonical key hash stores may use for NULL-safe uniqueness.
DetailStore
¶
Bases: ABC
Persistence axis for binding-declared named detail tables.
Every read takes accepted_lineage_ids, the lineages whose rows count, in preference order (the current
run's lineage first, as :meth:~altar.results.ScoreReusePolicy.ordered_ids orders them). The empty default
places no lineage restriction: rows of every lineage stored under the model count.
close
¶
Release the connections and clients this store opened itself.
The default does nothing. A store that opens a connection or creates a client overrides it, and a second
call must do nothing. A client passed to the constructor stays open: whoever created it closes it.
altar.run closes the stores it builds from a run spec when the run ends.
ensure_table
abstractmethod
async
¶
ensure_table(
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
schema: DetailSchema,
lineage_id: str | None = None,
) -> None
Create or validate the table of one lineage without silently changing its schema or plugin identity.
lineage_id is the score lineage of the producing run, the ID its primary rows are stored under; it
defaults to the lineage :meth:~altar.results.ScoreLineage.from_run generates for plugin_identity.
get_missing_keys
abstractmethod
async
¶
get_missing_keys(
*,
model_id: str,
detail_name: str,
keys: Sequence[DetailRowKey],
accepted_lineage_ids: Sequence[str] = (),
) -> set[DetailRowKey]
Return requested logical keys not persisted for this model and table under an accepted lineage.
get_missing_variants
async
¶
get_missing_variants(
*,
model_id: str,
detail_name: str,
variant_ids: Sequence[str],
accepted_lineage_ids: Sequence[str] = (),
) -> set[str]
Return requested parent variants with no persisted row in this model's table under an accepted lineage.
A detail table is one-to-many, so the logical keys a variant will produce are unknown before inference. This parent-level presence query lets cache-miss preparation treat a variant with no stored detail rows as incomplete without enumerating keys. It does not prove that a variant with some rows has all of them; callers combine it with the engine's details-before-primary write order.
The default streams :meth:read_rows and is correct for any adapter. Adapters should override it with
an indexed existence query so that wide tables (thousands of tracks per variant) are not read.
add_rows
abstractmethod
async
¶
add_rows(
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
schema: DetailSchema,
rows: Sequence[Mapping[str, object]],
lineage_id: str | None = None,
) -> InsertResult
Persist schema-shaped rows under one lineage and report adapter-side duplicate skips.
lineage_id is resolved as :meth:ensure_table resolves it.
read_rows
abstractmethod
async
¶
read_rows(
*,
model_id: str,
detail_name: str,
variant_ids: Sequence[str] | None = None,
accepted_lineage_ids: Sequence[str] = (),
) -> AsyncIterator[VariantRecord]
Stream stored rows, optionally restricted to parent variants and to accepted lineages.
With accepted lineages, a variant's rows come from the first lineage in accepted_lineage_ids that
holds rows for it, as primary reads choose one lineage per variant. Rows of a later lineage are not
read for that variant, so two producers' rows never mix. With none, rows of every lineage are read.
describe_table
abstractmethod
async
¶
describe_table(
model_id: str, detail_name: str, *, lineage_id: str
) -> StoredDetailTable | None
Return durable schema/plugin provenance of one lineage's table, or None when it is absent.
StoredDetailTable
dataclass
¶
StoredDetailTable(
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
schema: DetailSchema,
lineage_id: str,
)
Persisted scientific and runtime identity for one model's named detail table under one lineage.
lineage_id is the score lineage of the run that produced the table: the ID its primary rows carry.
plugin_identity is one representative exact identity of that lineage.
supplies
¶
supplies(schema: DetailSchema) -> bool
Whether this table's rows can stand in for rows of schema when its lineage is accepted for reuse.
Accepting a lineage is the caller's scientific decision about the producer; this is the structural half: the stored table must be the same named table, at the same declared version, with the same row identity and field meanings. Labels and descriptions do not matter.
DisplayRider
dataclass
¶
DisplayRider(order_by: str, columns: Sequence[RiderColumn])
Configuration for a variant's display columns.
It picks the single model-score row that sorts first by order_by (over the scored columns) and emits columns
from it.
This renders to two SQL steps. The first runs ROW_NUMBER() OVER (PARTITION BY variant_id ORDER BY <order_by>,
<model position>), with is_prioritized read as FALSE when a decision is missing; the second keeps
rank = 1 and evaluates the named column expressions on it. It uses ROW_NUMBER rather than
ARRAY_AGG(… LIMIT 1)[OFFSET(0)] so it runs on the BigQuery emulator.
order_by: a SQLORDER BYfragment that selects which of a variant's rows to emit from — for example"is_prioritized DESC, ABS(logfc) DESC"(prioritized models first, then largest|logFC|).columns: the ordered output columns emitted from the picked row.
RiderColumn
dataclass
¶
One output column of a display rider.
name is the emitted column, and expr is a SQL expression evaluated over the picked score row's columns (the
score columns plus is_prioritized, model_id, and model_name).
For example, RiderColumn("most_active_celltype", "model_name") and
RiderColumn("most_active_celltype_logfc", "logfc") emit the picked model's display name and its logFC.
The store keeps model_name among the scored columns, so expr can be a bare column reference rather
than a CASE model_id WHEN … THEN name lookup.
dtype is the column's scalar type in typed exports (the result buckets, Parquet, the column manifest). A bare
column reference infers it (rider_column_dtypes); any other expression must declare it.
ColumnManifestEntry
dataclass
¶
ColumnManifestEntry(
column: str,
kind: ColumnKind,
field: str,
dtype: str | None,
repeated: bool = False,
label: str | None = None,
source: str | None = None,
model_id: str | None = None,
model_name: str | None = None,
plugin_id: str | None = None,
plugin_version: str | None = None,
result_schema_version: str | None = None,
result_schema_hash: str | None = None,
)
One column of a wide export, as wide_column_manifest describes it (provisional).
columnis the column's name in the file, andkindwhat it holds:variant_key(variant_id),prioritization(the variant'sprioritizedand its two reason lists, or one model'sprioritized),display(a warehouse display rider's column),annotation(a source's column),score(a model's score field) orlineage(a model'sscore_lineage_id).fieldis the name the value has outside the wide file: a score field's own name, such aslog2fc, which models that share a result schema share; an annotation's column name; or the column's name otherwise.dtypeis the logical type,float,int,bool,str,jsonorstruct, andrepeatedsays the column is a list of it. A display rider's column has its type when the layout knows it (a result-bucket export) and None otherwise. A TSV holds a repeated, struct orjsonvalue as JSON text.labelis a score or annotation column's display label.sourcenames an annotation column's source.- The model fields are set on a model's columns:
model_id(the model key),model_name,plugin_id,plugin_version, and theresult_schema_versionandresult_schema_hashof the plugin's result schema.
to_dict
¶
Return the entry as JSON-compatible values, keyed in COLUMN_MANIFEST_FIELDS order.
WideExportLayout
dataclass
¶
WideExportLayout(
columns: tuple[str, ...],
rider_columns: tuple[str, ...],
annotation_columns: tuple[AnnotationColumn, ...],
model_blocks: tuple[ModelExportBlock, ...],
score_lineages: bool,
annotation_source_names: tuple[str, ...] = (),
rider_dtypes: tuple[ScalarDtype, ...] = (),
)
The column layout of the wide exports, built by wide_layout.
columns is every column in order. row renders one MaterializedVariant into that layout with logical
values: an absent annotation and every field of a model that did not score the variant are None. The
writers encode those values per format. annotation_source_names names the source of each of
annotation_columns, in the same order. rider_dtypes is the scalar type of each of rider_columns when it
is known (a layout built for result buckets knows them), and empty otherwise.
json_text_columns
¶
Return the annotation columns a CSV cell holds as JSON text: repeated, struct, and json columns.
DescriptorTableNotProvisioned
¶
Bases: CapabilityNotSupported
Raised when a store must record a lineage descriptor but its host has not provisioned the descriptor table.
A store that never creates the table implicitly (BigQuery) raises this instead of a bare
CapabilityNotSupported, so a caller can tell a table waiting to be provisioned from a store that keeps no
descriptors at all. table names the table to provision, or to create with ensure_descriptor_table().
DescriptorTableNotProvisionedWarning
¶
Bases: UserWarning
Emitted when an engine path cannot record a run's lineage descriptor because its table is not provisioned.
The run proceeds and its scores are written, but the lineage takes part in read-time equivalence only once its
descriptor is recorded. Provision the named table, or call ensure_descriptor_table(), and score or prepare
the configuration again to record it. prepare_scoring_batches, run_scoring, and ingest_scoring_results
emit it at most once per store instance and table.
LongModelInfo
dataclass
¶
LongModelInfo(
lineage_id: str | None = None,
registry: str | None = None,
registry_id: str | None = None,
registry_entry_sha256: str | None = None,
)
What a caller knows about one model beyond the layout, for its row of the models table (provisional).
lineage_id is the lineage the model's run requested; the rows themselves carry the lineage their scores came
from in score_lineage_id. registry, registry_id and registry_entry_sha256 name the registry entry the
model came from, when it came from one.
LongScoreTable
dataclass
¶
LongScoreTable(
name: str,
plugin_id: str,
result_schema_version: str,
model_ids: tuple[str, ...],
fields: tuple[ExportField, ...],
)
One score table of the long export: the models whose plugin and result schema version it holds (provisional).
name is the table's directory under scores/. fields are its score columns, each named by its own field
name, in the order the models declare them; a field one model of the table lacks is NULL in that model's rows.
AnnotationPrioritizer
dataclass
¶
AnnotationPrioritizer(name: str, predicate: Predicate)
A reference source that can prioritize a variant without a model, such as AlphaMissense.
It pairs a name with a Predicate that decides whether the variant is prioritized, evaluated over
the variant's annotation columns. Unlike a model, it has no model_id and produces no score entry.
When its predicate is true, it only contributes to the overall prioritized flag.
annotation_prioritizers_from builds these from the configured sources.
MaterializedVariant
dataclass
¶
MaterializedVariant(
variant_id: str,
model_scores: list[dict[str, Any]],
prioritized: bool,
prioritized_model_ids: list[str],
prioritized_source_names: list[str],
plugin_identities: dict[str, dict[str, Any]],
score_lineages: dict[str, str] = dict(),
annotations: dict[str, Any] = dict(),
)
The per-variant summary produced by materialize_variant.
It holds only the fields that are the same across every model architecture. The optional display values — the headline score magnitude and the most-active cell type — are decided by each plugin and are not fields here.
annotations maps each declared annotation column that has a value for this variant to that value, merged
across every configured source. A column with no value is absent rather than None: a source with no row
for the variant, a null cell, and an empty repeated value all leave the column out (BigQuery returns a
null array as an empty one, so no store can tell those apart). A variant no source annotated gets {}.
Values keep their logical type on every store: a repeated column is a list and a struct column a dict
(a list of dicts when repeated), never the JSON text a backend such as SQLite stores them as. Values
nested inside a struct or list are kept as the source returned them.
ModelScore
dataclass
¶
ModelScore(model: ScoredModel, score: VariantRecord)
One model's raw score for a variant, keyed by the model's score_columns (for example logfc or
in_peak).
ScoredModel
dataclass
¶
ScoredModel(
model_id: str,
model_name: str,
plugin: ModelResults,
plugin_identity: PluginRunIdentity,
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
lineage_descriptor: LineageDescriptor | None = None,
)
A model that scored the job's variants, together with its architecture plugin.
plugin is typed as ModelResults, the smaller interface that exposes only the methods for reading
results. Materialization never needs the model-building methods (build_*), so it cannot call them by
accident. A full ModelPlugin also satisfies this type.
lineage_descriptor
class-attribute
instance-attribute
¶
lineage_descriptor: LineageDescriptor | None = None
The run's lineage descriptor, so a compact-lineage store reads equivalent lineages before it is recorded.
equivalence
property
¶
equivalence: ModelPlugin | None
The plugin whose equivalence rules resolve this model's lineages: plugin, when it is a ModelPlugin.
ExportDuplicateRowsError
¶
ExportDuplicateRowsError(
row_count: int,
variant_count: int,
*,
where: str = "the export's result table",
)
Bases: ValueError
An export holds more than one row for some variant, where its contract is one row per variant (provisional).
row_count rows hold variant_count distinct variants. BigQueryScoreStore checks its result table right after
building it and raises this before any EXPORT DATA job or Read API session, so nothing has been written.
ResultBuckets raises it for any store's bucket that repeats a variant, before a writer renders that bucket. The
usual cause is an annotation source that returns several rows for one variant, such as a reference table loaded
twice or a relation= that does not aggregate per variant. The stores collapse each source to one row per
variant, so this means a store's query or a source broke that rule.
ExportScanLimitExceeded
¶
Bases: RuntimeError
A store refused an export whose estimated scan exceeds the configured maximum_bytes_billed.
Raised before any export job runs, so nothing is billed. estimated_bytes is the store's estimate (for
BigQuery a dry run, an upper bound on a clustered table) and maximum_bytes_billed the configured limit. A
host catches it to record the export as unavailable rather than fail the job.
ResultBucket
dataclass
¶
ResultBucket(
descriptor: ResultBucketDescriptor, table: Table
)
One bucket's rows (provisional): a table with result_bucket_schema, its rows in no particular order.
sorted_table
¶
Return the bucket's rows sorted by variant_ordinal, the order every writer renders them in.
ResultBucketDescriptor
dataclass
¶
ResultBucketDescriptor(
index: int,
first_ordinal: int,
last_ordinal: int,
row_count: int,
sort_key: tuple[str, ...] = (VARIANT_ORDINAL,),
schema_version: int = RESULT_BUCKET_SCHEMA_VERSION,
)
One bucket of an export (provisional): its number, the ordinals it covers, and its row count.
first_ordinal and last_ordinal are inclusive, and row_count is last_ordinal - first_ordinal + 1: every
variant of the export has exactly one row. sort_key names the column a writer sorts the bucket's rows by.
ResultBuckets
¶
ResultBuckets(
layout: WideExportLayout,
descriptors: Sequence[ResultBucketDescriptor],
load: BucketLoader | None = None,
*,
models: Sequence[ScoredModel] = (),
reader: BucketReader | None = None,
)
The buckets of one export, as a store produces them (provisional).
layout is the export's wide layout (with the rider's columns and types), schema the bucket schema, and
descriptors the buckets in genomic order. Iterating yields each ResultBucket in that order, reading one at
a time through the producer's loader, which a later iteration calls again; a writer that drops each bucket
before asking for the next holds one bucket. Each table is cast to schema, an empty repeated annotation is
turned into null, a table that repeats a variant_ordinal raises ExportDuplicateRowsError, and one whose row
count or ordinals do not match its descriptor raises ValueError.
models are the export's models, in order.
A producer passes load, a loader every iteration shares, or reader, which opens a loader of its own for each
iteration and closes it when the iteration ends: a producer whose reading keeps state, such as the local copies
of staged files, uses it so that several writers can iterate the same buckets at once.
DescriptorAttachmentFailure
dataclass
¶
DescriptorAttachmentFailure(
identity: PluginRunIdentity,
descriptor: LineageDescriptor,
reason: str,
)
One (identity, descriptor) pair ScoreStore.attach_lineage_descriptors did not record, and why.
DescriptorAttachmentReport
dataclass
¶
DescriptorAttachmentReport(
attached: tuple[str, ...] = (),
unregistered: tuple[str, ...] = (),
failed: tuple[DescriptorAttachmentFailure, ...] = (),
)
What ScoreStore.attach_lineage_descriptors did, by lineage ID in input order.
attached lineages are registered and now carry their descriptor; unregistered ones had their
descriptor recorded but have no catalog entry yet, so they join equivalence once registered; failed
pairs did not verify or conflict with a recorded descriptor, and recorded nothing.
Resolution
¶
Bases: tuple[ScoreLineage, ...]
The lineages a read accepts, in preference order: a tuple of ScoreLineage with how each was chosen.
The current lineage comes first, then the registered lineages the binding declares equivalent (by lineage
ID), then the policy's accepted lineages. steps holds each lineage's ResolutionStep, and rules_id
the plugin's equivalence_rules_id when the equivalence step ran, else None. Freeze it with
ScoreReusePolicy.exactly(resolution.lineage_ids).
steps
property
¶
steps: tuple[ResolutionStep, ...]
How each lineage was chosen, aligned with the tuple.
rules_id
property
¶
The equivalence_rules_id the equivalence step applied, or None when it did not run.
ResolutionStep
¶
Bases: StrEnum
How a lineage entered a Resolution.
ScoreLineage
dataclass
¶
ScoreLineage(
lineage_id: str,
plugin_id: str,
columns: tuple[ScoreColumn, ...],
metadata_json: str = "{}",
run_identity: PluginRunIdentity | None = None,
)
An immutable catalog entry referenced by a compact ID on score rows.
An imported lineage may cover many models and input genomes: those remain
separate row keys. metadata_json holds publisher/operator declarations,
including any model-specific settings. Historical imports need no invented
Altar run identity. Registering a lineage does not by itself accept it for reuse.
from_run
classmethod
¶
from_run(
identity: PluginRunIdentity,
columns: Sequence[ScoreColumn],
) -> ScoreLineage
Key the lineage on the run's scientific projection; keep the exact identity for provenance.
The ID hashes PluginRunIdentity.scientific_dict(), so two runs share a lineage exactly when
their results are scientifically interchangeable (see same_lineage). Exact provenance (resource
locations, the declared configuration serialization, framework versions) never decides reuse and
is never repeated on score rows. A lineage store keeps one deterministic representative run identity
as the catalog's run_identity and records every distinct exact identity that registers the lineage
to write scores; ScoreStore.read_lineage_provenance returns that set.
same_lineage
¶
same_lineage(other: ScoreLineage) -> bool
Whether other declares this lineage, so a catalog holding either accepts the other.
Both must name the same ID, plugin, output columns, and metadata. A generated lineage (one with a
run_identity) is defined by its run's PluginRunIdentity.scientific_dict(), so declarations
from runs that differ only in exact provenance agree. An imported lineage has no run identity and
agrees only with an identical declaration; it never agrees with a generated one.
validate_outputs
¶
validate_outputs(
plugin_id: str, columns: Sequence[ScoreColumn]
) -> None
Acceptance cannot cross plugins or substitute missing/incompatible outputs.
ScoreReusePolicy
dataclass
¶
Additional acceptable lineages, in preference order after the current run.
This is an explicit scientific decision by the caller, scoped to the model and genome being queried. It is neither inferred from version numbers nor transitive. An empty policy accepts only the current run, plus the lineages its binding declares equivalent when the store is given the plugin and a descriptor. Acceptance changes reads, never the lineage stamped on newly computed scores.
An exact policy, built with :meth:exactly, instead lists every lineage to read, the current one
first, and resolves to exactly that list.
exactly
classmethod
¶
Read exactly lineage_ids, in that order of preference, such as a frozen Resolution.
The first ID must be the run's current lineage, which need not be registered; every other ID must be.
Equivalent lineages are not added. A Resolution passes its lineage_ids.
ordered_ids
¶
ordered_ids(current: ScoreLineage) -> tuple[str, ...]
Return the lineage IDs this policy names for a run whose current lineage is current, current first.
StoredScore
dataclass
¶
StoredScore(values: VariantRecord, lineage: ScoreLineage)
A selected score and the lineage it was stored under, including imported provenance.
In compact-lineage mode lineage.run_identity is the lineage's reference identity, one deterministic
representative of the runs that wrote under it; a row is not attributed to the exact identity that wrote
it. ScoreStore.read_lineage_provenance returns the set of exact identities that wrote under the lineage.
SchemaCollisionError
¶
Bases: ValueError
Raised when independently owned plugin columns cannot be composed safely.
PrimaryScoreWriter
¶
Bases: ABC
The writer one ScoreStore.primary_write unit yields: it accepts the unit's primary rows batch by batch.
Every batch belongs to the model, run identity, and columns the unit was opened with. add takes rows in
the shape ScoreStore.add_scores takes and returns an InsertResult for that batch: added counts the
rows the unit accepted, skipped the rows the store declined because it already held them. For a store
whose unit is atomic, the counted rows become visible only when the unit commits, and none of them do if
it aborts.
add
abstractmethod
async
¶
add(rows: Tabular) -> InsertResult
Stage or write one batch of primary rows, and return its counts.
ScoreStore
¶
Bases: ABC
Base class for a store that writes and reads per-model variant scores.
Stores register under altar.score_stores and are resolved by name. name is the registry key, for
example "sqlite" or "bigquery".
A note on the parameter types. At the call sites, rows (and, for a warehouse store, variant_ids) may
be pandas objects, but core itself does not depend on pandas, so they are typed as Tabular (a name for
Any documented in types); the conformance suite is what actually pins their shape. The read methods return the portable
VariantRecord instead. The columns a store persists come from plugin.score_columns(), which is
the one place the score-column set is defined.
atomic_primary_write
class-attribute
¶
Whether primary_write publishes all of a unit's rows on commit and none of them on abort.
lineage_provenance_table_id
property
¶
Return the table that records each exact run identity under a lineage, or None without a catalog.
A host that provisions or checks the lineage tables itself reads the name here. Stores that keep a lineage catalog override this; the base store keeps none.
lineage_descriptor_table_id
property
¶
Return the table that holds one LineageDescriptor per lineage, or None without a catalog.
Stores that keep a lineage catalog override this; the base store keeps none.
close
¶
Release the connections and clients this store opened itself.
The default does nothing. A store that opens a connection or creates a client overrides it, and a second
call must do nothing. A client passed to the constructor stays open: whoever created it closes it.
altar.run closes the stores it builds from a run spec when the run ends.
register_score_lineage
async
¶
register_score_lineage(
lineage: ScoreLineage,
*,
lineage_descriptor: LineageDescriptor | None = None,
) -> None
Register immutable metadata before importing/backfilling its compact row ID.
This does not rewrite scores or accept the lineage for any request.
Re-registering a declaration that ScoreLineage.same_lineage accepts is idempotent and keeps the
catalog entry; any other change to an existing ID fails. A generated lineage may pass its
lineage_descriptor, which is verified against its run identity and recorded first.
ensure_descriptor_table
async
¶
Explicitly create this store's lineage descriptor sidecar if it is missing, and check its columns.
A host may provision the sidecar itself instead; extra nullable columns are accepted. SQLite also creates it on the first descriptor it records; BigQuery never creates it except through this call.
resolve_lineages
async
¶
resolve_lineages(
plugin_identity: PluginRunIdentity,
columns: Sequence[ScoreColumn],
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
*,
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
) -> Resolution
Return the lineages a read under plugin_identity accepts, in preference order, without reading scores.
The Resolution lists the current lineage, then (when equivalence, the run's ModelPlugin, is
given) every registered lineage whose descriptor that plugin's equivalence_projection makes equal to
the current one, then reuse_policy's accepted lineages. The current descriptor is
lineage_descriptor, verified against the identity, or else the recorded one. Freeze a resolution with
ScoreReusePolicy.exactly(resolution.lineage_ids) to read it back unchanged later.
record_lineage_descriptor
async
¶
record_lineage_descriptor(
plugin_identity: PluginRunIdentity,
descriptor: LineageDescriptor,
) -> None
Verify descriptor against plugin_identity and record it for the run's lineage.
Recording needs no registration, so a host records the descriptor at admission and the lineage joins equivalence when a write registers it. Recording again is a no-op; a different descriptor is refused.
attach_lineage_descriptors
async
¶
attach_lineage_descriptors(
items: Iterable[
tuple[PluginRunIdentity, LineageDescriptor]
],
) -> DescriptorAttachmentReport
Verify and record descriptors for existing lineages, such as every registration a host holds.
Every pair whose descriptor verifies against its identity is recorded, registered or not; the report separates registered lineages from unregistered ones and lists the pairs that failed.
lineages_without_descriptors
async
¶
Return the registered lineages of plugin_id that have a run identity but no recorded descriptor.
A descriptor backfill is complete when each one is attached or known to be unattachable.
read_lineage_provenance
async
¶
read_lineage_provenance(
lineage_id: str,
) -> tuple[PluginRunIdentity, ...]
Return the set of distinct exact run identities under a lineage, oldest record first.
A generated lineage is keyed on its run's scientific projection, so runs that differ only in exact
provenance (resource locations, score-neutral configuration serialization, framework API versions)
share it and reuse each other's scores. The catalog keeps one deterministic reference run_identity
per lineage, always included here; each run that registers to write scores under the lineage (even if
all its rows are skipped as duplicates) or registers its generated lineage explicitly is recorded once.
Rows are attributed to the lineage, not to one of these identities. An imported lineage, or an ID
this store never registered, has none.
get_missing_scores
async
¶
get_missing_scores(
variant_ids: Sequence[str],
model_id: str,
*,
plugin_identity: PluginRunIdentity,
columns: Sequence[ScoreColumn],
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
) -> list[str]
Return candidate misses under a model-, genome-, and lineage-scoped policy.
With compact lineages, lineage_descriptor and equivalence resolve as resolve_lineages does.
read_selected_scores
async
¶
read_selected_scores(
variant_ids: Sequence[str],
model_id: str,
*,
plugin_identity: PluginRunIdentity,
columns: Sequence[ScoreColumn],
reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
lineage_descriptor: LineageDescriptor | None = None,
equivalence: ModelPlugin | None = None,
) -> dict[str, StoredScore]
Select one whole row per variant, retaining its actual lineage.
Current-run results win; otherwise the first available lineage in resolution order wins (see
resolve_lineages, whose lineage_descriptor and equivalence this takes).
Output presence is a catalog declaration, not a test for non-NULL values.
required_row_identity
¶
Return typed row-identity fields this adapter requires in addition to variant_id.
The common engine preserves only explicitly declared portable identity fields. Persistence-only
stores normally need no decomposed locus and inherit the empty mapping. Warehouse stores can require
the portable chr/pos/ref/alt locus without teaching a model binding about that backend:
the engine derives omitted locus fields from the canonical variant_id and verifies emitted ones.
Portable fields keep their portable dtypes; other required fields must be emitted by the codec.
ensure_schema
abstractmethod
async
¶
ensure_schema(columns: list[ScoreColumn]) -> None
Additively set up the backing store to hold columns plus shared variant identity.
Existing model schemas remain intact. Reusing a column name is valid when its storage definition
agrees with the existing one; display-label changes do not alter stored values. Both reference stores
take the column set from score_columns().
add_scores
abstractmethod
async
¶
add_scores(
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
rows: Tabular,
columns: list[ScoreColumn],
) -> InsertResult
Append one model's per-variant scores. rows holds the columns plus the variant-identity columns.
Returns an InsertResult: added is the number of rows this call newly persisted, and skipped
is the number the store declined because it already held a score for that (model_id, variant_id).
A score is a finite value, or None when it is missing. A store raises ValueError for NaN or an
infinity, before it persists any row of the call.
Idempotency is adapter-specific and NOT guaranteed by this contract. A store MAY dedup on re-add
(the sqlite store uses INSERT OR IGNORE, so re-adding a variant lands in skipped) or MAY append
unconditionally (the BigQuery store uses WRITE_APPEND, always reporting skipped=0 and
physically duplicating a re-added row). Callers therefore dedup upstream — get_unscored_variants
is the gate that keeps already-scored variants out of rows — and must not rely on add_scores
itself to be idempotent.
primary_write
¶
primary_write(
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
columns: list[ScoreColumn],
lineage_descriptor: LineageDescriptor | None = None,
) -> AbstractAsyncContextManager[PrimaryScoreWriter]
Open one staged write unit for a run's primary rows: async with store.primary_write(...) as writer.
The unit yields a PrimaryScoreWriter whose add(rows) takes one batch at a time. A clean exit
commits the unit; an exception aborts it and propagates. The scoring engine writes every primary row of
one ingest through one unit, after the run's details and artifacts, so the primary rows stay the
per-variant commit marker.
The default is a pass-through: each add is one immediate add_scores call, so a batch is durable as
soon as it returns and an abort leaves the batches already added in place. It is not atomic, and
atomic_primary_write stays False. A store that can stage the batches of a whole unit and publish
them together (one database transaction; a staging table promoted on commit and dropped on abort)
overrides this method and sets atomic_primary_write = True; where the staged rows live is the
store's concern, never the caller's. A store that subclasses another and changes how add_scores
writes must override this method too, or inherit a unit that bypasses its add_scores.
lineage_descriptor, the run's LineageDescriptor, is verified when the unit opens; a compact-lineage
store records it when the unit registers the lineage. Other stores keep no descriptors.
read_run_identity
abstractmethod
async
¶
read_run_identity(
model_id: str,
) -> PluginRunIdentity | None
Return recorded provenance compatible with model_id scores, or None when none exist.
A store may retain more than one exact run identity when compatible wrapper releases incrementally populated the model cache; this method returns one representative immutable identity. The engine compares its role-specific scientific compatibility before treating existing rows as cache hits.
get_unscored_variants
abstractmethod
async
¶
Return the variant_ids this store has no score for under model_id, so already-scored variants
are not scored again. Mirrors AnnotationSource.get_unannotated_variants, which names the same input
the same way.
read_score
async
¶
Read one model's stored score columns for one variant. Return None if this store holds no score
for that (model_id, variant_id).
This is the only read a persistence-only store has to implement. The materialize driver calls it,
through read_scores, to build each variant's list of ModelScore. The return value maps the
plugin's score_columns keys to their values, with boolean columns decoded back from whatever the
store used to save them. The method is async so a database-backed store can await its round trip
instead of blocking the event loop. The default implementation raises, so a store that overrides
materialize with a single query (such as BigQuery) does not have to provide it.
read_scores
async
¶
Read every stored score for model_id among variant_ids, keyed by variant_id. Variants this
store has no score for are left out of the returned mapping.
The materialize driver calls this rather than read_score, so it does one read per model instead
of one read per (model, variant) pair. With N variants and M models, that is M reads rather than
M×N single-row round trips on the event loop. The default implementation here just calls read_score
for each id. A store backed by a query engine should override it with a single
variant_id IN (...) fetch, as SqliteScoreStore does. A persistence-only store can implement
only read_score and inherit this.
materialize
async
¶
materialize(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
) -> AsyncIterator[MaterializedVariant]
Produce one per-variant summary for each distinct id in variant_ids, in ascending variant_id order.
This is the default driver that persistence-only stores inherit.
Every store yields the same rows in the same order: exactly one MaterializedVariant per distinct
requested variant — a repeated id is materialized once — ordered by canonical variant_id (plain
string order, as SQL ORDER BY variant_id sorts it). Request order is not kept: a warehouse store
streams its result sorted by variant_id so it can group each variant's rows without buffering the
job, and a relation-scoped request has no request order to keep. A non-canonical id raises
VariantIdentityError.
Only the sources that serve the models' assembly are joined (see applicable_annotation_sources): a
source that declares another assembly than the one the models were scored on never annotates these
variants, and a dependency only such a source supplies reads as missing.
It walks the sorted variants in chunks of _MATERIALIZE_CHUNK_VARIANTS (10,000). For each chunk it loads
and merges the chunk's annotations with fetch_annotations, collects each model's scores for the chunk
through read_selected_scores, and passes each variant's to materialize_variant, which does the join and
prioritization the same way for every store. It yields one MaterializedVariant per variant. A
persistence-only store inherits this method unchanged and implements only read_score.
The method is an async generator: a job with millions of variants must not hold the whole result in
memory, so callers async for over the yielded variants, and only one chunk's annotations and scores
are held at a time. A source or store error in a later chunk is therefore raised after the earlier
chunks' rows were yielded. Each yielded MaterializedVariant carries
model_scores, the prioritized / prioritized_model_ids / prioritized_source_names rollup, and
the variant's annotations. Every store fills annotations the same way: the declared columns that have
a value, with their logical types (see MaterializedVariant). The most-active-cell-type pick and the
headline sort value are computed by a separate plugin-declared display rollup, not here.
A warehouse store that can do the join and prioritization in one query overrides this method; see
BigQueryScoreStore. Anything specific to such a backend — for example passing a temporary-table
reference when the id list is too long for one request — lives on that override, not on this driver.
export_download
async
¶
export_download(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
destination_uri: str,
storage: Storage | None = None,
prioritized_only: bool = False,
) -> str
Write every requested variant as one flat, wide, tab-delimited download under destination_uri (a
prefix) and return that URI.
This produces a different artifact from materialize. It covers every requested variant, not only the
prioritized ones, and lays out each variant as one row with the per-model score columns side by side.
Its consumer is a file download, not the database rows. The columns are the wide layout of
altar.plugins.data_lake.export (wide_layout), the same on every store. prioritized_only keeps
only the prioritized variants, the prioritized download.
The default implementation materializes and writes the file with write_wide_csv: one gzipped part,
<destination_uri>/part-000000000000.csv.gz, written through storage (the local filesystem when
None). A backend that can export in bulk overrides it: BigQueryScoreStore runs one EXPORT DATA
job into a gs:// prefix, with the same columns, and does not use storage. Anything specific to
that backend, such as scoping the export to a temporary-table reference, lives on the override rather
than in this base signature; see materialize.
export_prioritized
async
¶
export_prioritized(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
destination_uri: str,
storage: Storage | None = None,
) -> str
Write only the prioritized variants among variant_ids as NDJSON under destination_uri (a prefix)
and return that URI.
This is the database branch of the two exports: each line is one MaterializedVariant as JSON, with
model_scores nested (see prioritized_record), so a loader copies a part into a database and decodes
each row with one json.loads. That database load is the caller's step, not this method.
The default implementation materializes and writes <destination_uri>/part-000000000000.json.gz
through storage (the local filesystem when None) with write_prioritized_ndjson.
BigQueryScoreStore overrides it with one EXPORT DATA job that aggregates the same records in SQL.
result_buckets
¶
result_buckets(
*,
variant_ids: Sequence[str],
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
rider: DisplayRider | None = None,
bucket_rows: int | None = None,
bucket_bytes: int | None = None,
) -> AbstractAsyncContextManager[ResultBuckets]
Produce the export of variant_ids as result buckets: async with store.result_buckets(...) as buckets.
The buckets (altar.plugins.data_lake.result_buckets) hold one row per distinct requested variant, in
genomic order, with its annotations once and its models' scores nested in model_scores; the
altar.plugins.data_lake.bucket_export writers render them to every export layout into any Storage. The
scope and values are materialize's. rider, a DisplayRider, adds its columns, computed per variant.
bucket_rows sets the variants per bucket, or bucket_bytes a target size (default 128 MiB) that
estimated_variant_bytes turns into rows the same way on every store. The context manager releases what
the store created for the export when the block exits.
The default reads each bucket with materialize over the bucket's variants, so a persistence-only store
exports without implementing anything, and evaluates the rider with SQLite (evaluate_rider). A warehouse
store overrides it with its own query (BigQueryScoreStore).
SqliteScoreStore
¶
Bases: ScoreStore
Score store backed by SQLite. It uses one table, variant_scores, keyed by (model_id, variant_id),
with one column per plugin score column. New columns are added by ensure_schema without dropping any.
primary_write
¶
primary_write(
*,
model_id: str,
model_name: str,
plugin_identity: PluginRunIdentity,
columns: list[ScoreColumn],
lineage_descriptor: LineageDescriptor | None = None,
) -> AbstractAsyncContextManager[PrimaryScoreWriter]
Write every batch of the unit in one SQLite transaction: a commit publishes all of them, an abort none.
Schema changes commit on their own, so the unit applies them before it begins the transaction. The
model's schema and run identity (or, with compact lineages, the lineage declaration and its
provenance) are registered inside it, by the first batch, as add_scores registers them, so an
aborted unit leaves no trace of its run. While the unit is open, reads on this store see its uncommitted
rows, and every other write through this store raises rather than committing the unit early. A
lineage_descriptor is verified when the unit opens and, with compact lineages, recorded with the
lineage inside the same transaction.
get_unscored_variants
async
¶
Return the variant_ids this store holds no score for under model_id, in input order.
It looks the IDs up in variant_id IN (...) queries of a few hundred IDs each, as read_scores does, so
a call reads only the requested keys however many scores the model already has.
read_score
async
¶
Return one model's stored score for a variant, or None if there is none. Calls the bulk
read_scores so that the row-decoding logic lives in one place.
read_scores
async
¶
Return the stored scores for model_id across the given variant_ids, keyed by variant id.
This fetches them in variant_id IN (...) queries of a few hundred ids each, so the inherited
materialize driver does one bulk read per model instead of one read per (model, variant) pair, and a
long id list stays under SQLite's host-parameter limit. Bool columns are converted
back from their 0/1 storage. The model-specific schema is persisted alongside the score rows, so
ensuring another plugin's schema cannot change the shape of this read.
close
¶
Close the store's SQLite connection; a second call does nothing.
Close a file-backed store when you are done with it, or open it in a with block, which closes it on exit.
SqliteDetailStore
¶
Bases: DetailStore
Lossless named detail storage using schema-independent SQLite tables.
Rows are kept as canonical JSON while their parent variant, lineage, and logical-key digest are indexed. This reference layout can accept any binding schema without adding physical columns or coupling a backend to a model. One metadata row describes each lineage's table, and a provenance sidecar retains every compatible exact run identity used to populate it. A production adapter may use native typed columns while preserving the same contract.
StorageArtifactStore
¶
StorageArtifactStore(
storage: Storage,
root_uri: str,
index_path: str = ":memory:",
)
Bases: ArtifactStore
Artifact files in a Storage tree under root_uri; their records in a SQLite index at index_path.
One metadata row describes each lineage's artifact, a provenance sidecar retains every compatible exact run
identity that wrote under it, and a variant table indexes which variants each file covers so coverage
queries never read the files. index_path may be ":memory:" for tests.
locate
async
¶
locate(record: ArtifactRecord) -> str
Return the record's URI; the bytes are readable through this store's storage.
InsertResult
dataclass
¶
The result of a score or annotation write: how many rows were added and how many were skipped
because the store already held them.
Both data-lake roles return this — ScoreStore.add_scores and AnnotationSource.add_annotations — so
it lives in this shared module rather than in either role's own module. Whether a write dedups at all is
adapter-specific: an append-only store (for example the BigQuery score store) always reports
skipped=0, while a store that ignores re-adds (the sqlite stores) counts them under skipped. See
ScoreStore.add_scores for the idempotency contract.
ScoreColumn
dataclass
¶
One per-variant score column a model plugin produces.
Declaring the column here sets four things that must otherwise be kept in sync by hand: the score-store schema (table columns / DDL), the write-time validator, the wide-TSV column names, and the frontend column definitions.
It lives in this shared leaf module — not in model/plugin.py — because it is the schema the score
store consumes (ScoreStore.ensure_schema / add_scores); keeping it here lets the data-lake axis
name it without importing back into the model axis. model.plugin re-exports it for architecture
authors.
get_artifact_store_registry
¶
get_artifact_store_registry() -> PluginRegistry[
type[ArtifactStore]
]
Return the shared registry of artifact-store adapters.
require_artifact_coverage
¶
require_artifact_coverage(
schema: ArtifactSchema,
key: ArtifactKey,
variant_ids: Sequence[str],
) -> None
Reject a file whose key and coverage do not fit the artifact's declared scope.
A batch file's key names no variant, since the file covers its task's whole batch. A variant file's
key names the one variant the file covers, and variant_ids is exactly that variant. Every
:class:ArtifactStore applies this rule in :meth:~ArtifactStore.put, so a router that collected a
per-variant file as if it covered the batch, or the reverse, fails before anything is recorded.
render_long_export
async
¶
render_long_export(
buckets: ResultBuckets,
destination_uri: str,
*,
storage: Storage,
export_format: LongExportFormat = "parquet",
prioritized_only: bool = False,
model_info: Mapping[str, LongModelInfo] | None = None,
provenance: ProvenanceCallback | None = None,
) -> dict[str, str]
Write the buckets as the long layout under destination_uri, and return each table's directory.
The tables, files and values are those of write_long_export: models, variants and one scores/<table>
per result schema, each one part, Parquet or (export_format="tsv") gzipped TSV. The variants table also
carries the rider's columns, after prioritized, when the export has a rider. Rows are in genomic order, and a
score table's rows for one variant follow the export's model order. Each table is written to local scratch as
the buckets arrive and uploaded at the end.
render_occurrence_buckets
async
¶
render_occurrence_buckets(
buckets: ResultBuckets,
occurrence_parts: Sequence[str],
destination_uri: str,
*,
storage: Storage,
bucket_size: int,
) -> list[str]
Join the buckets' wide rows onto the ingest occurrence table, one Parquet part per record-ordinal bucket.
This is write_occurrence_buckets fed from result buckets, with the same altar-augmentation-buckets-v1
directory and the same columns (bucket_schema), rider columns included. The wide rows are spilled to a SQLite
file keyed by variant as the buckets arrive, then each occurrence's row is read back in record order.
render_prioritized_ndjson
async
¶
render_prioritized_ndjson(
buckets: ResultBuckets,
destination_uri: str,
*,
storage: Storage,
int64_as_string: bool = True,
) -> str
Write the prioritized variants as gzipped NDJSON, <destination_uri>/part-000000000000.json.gz.
Each line is the record BigQueryScoreStore.export_prioritized exports, field for field: variant_id, the
rider's columns, model_scores (each entry model_id, model_name, every score field of the export,
prioritized, and score_lineage_id on a compact-lineage export), prioritized_model_ids,
prioritized_source_names, each annotation column in its native shape, then prioritized. An entry's
model_name is the name the store holds with the score (the scores table's, on BigQuery). Records are in
genomic order, and each record's model_scores in the order of the export's models (BigQuery's own export
aggregates them in no set order).
Altar's NDJSON format: a missing repeated value is [], a NaN or infinite float is the string "NaN",
"Infinity" or "-Infinity" (JSON has no such numbers, and a database's JSON type refuses the bare tokens), with
int64_as_string (the default) an integer is a JSON string, a null field is written as null, and a float is its
shortest round-trip text. BigQuery's JSON export holds the same values in other text: it omits null fields and
writes floats to 15 or 17 significant digits. altar.testing.compare_prioritized_ndjson compares the two by
value.
render_wide_parquet
async
¶
render_wide_parquet(
buckets: ResultBuckets,
destination_uri: str,
*,
storage: Storage,
prioritized_only: bool = False,
provenance: ProvenanceCallback | None = None,
) -> str
Write the buckets as the wide table in Parquet, <destination_uri>/part-000000000000.parquet.
The schema is arrow_schema(buckets.layout), rider columns included. Each bucket is one row group, so the
bucket size sets the row-group size. With provenance, its mapping is stored as JSON text in the file's
key-value metadata under altar.provenance.
render_wide_tsv
async
¶
render_wide_tsv(
buckets: ResultBuckets,
destination_uri: str,
*,
storage: Storage,
prioritized_only: bool = False,
part: str = DOWNLOAD_PART,
provenance: ProvenanceCallback | None = None,
) -> str
Write the buckets as the gzipped, tab-delimited wide table <destination_uri>/<part>, and return the prefix.
The columns are buckets.layout.columns, with a header row, and the cells are those of write_wide_csv: NULL
is empty, a boolean true/false, a repeated, struct or json value its compact JSON text. Rows are in
genomic order. prioritized_only keeps only the prioritized variants. part defaults to the name a warehouse
export's first shard has; altar run passes TSV_PART. With provenance, its mapping is written to
<destination_uri>/_provenance.json after the last row.
get_detail_store_registry
¶
get_detail_store_registry() -> PluginRegistry[
type[DetailStore]
]
Return the shared registry of detail-store adapters.
column_manifest_tsv
¶
column_manifest_tsv(
entries: Iterable[ColumnManifestEntry],
) -> str
Render manifest entries as the text of _columns.tsv (provisional).
A header row of COLUMN_MANIFEST_FIELDS, then one tab-separated row per entry, quoted as the wide TSV quotes
its cells. An unset field is empty, and repeated is true or false.
export_col_alias
¶
Build the wide-export column name of one model field: the model's prefix plus the field's name.
The score-column suffix is preserved for consumers that group columns by score type.
export_column_prefix
¶
Build a model's stable, collision-resistant column prefix for the wide exports.
The readable token comes from model_name; a digest of the stable model_id distinguishes duplicate
names and names that sanitize to the same token. model_id is already required to be unique within a
materialization. The augmented-VCF writer reads this prefix from its models JSON as column_prefix.
wide_column_manifest
¶
wide_column_manifest(
layout: WideExportLayout,
) -> list[ColumnManifestEntry]
Describe every column of layout, one entry per column in layout.columns order (provisional).
It reads only the layout, so it describes any file with these columns: the portable writers' TSV and Parquet
files, an altar run export, and BigQueryScoreStore.export_download's TSV for the layout its
download_header names, whose display rider columns are entries of kind display.
wide_layout
¶
wide_layout(
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
*,
rider_columns: Sequence[str] = (),
score_lineages: bool = False,
rider_dtypes: Sequence[ScalarDtype] = (),
) -> WideExportLayout
Build the wide-export layout for models and annotation_sources (see the module docstring).
rider_columns are a display rider's output columns, placed after prioritized, and rider_dtypes, when
given, their scalar types in the same order.
score_lineages adds each model's score_lineage_id column, for a store in compact-lineage mode. Raises
SchemaCollisionError when two sources declare one column, a score column collides with an annotation
column, or any two export columns share a name.
write_column_manifest
async
¶
write_column_manifest(
layout: WideExportLayout,
destination_uri: str,
*,
storage: Storage,
) -> str
Write the column manifest of layout as <destination_uri>/_columns.tsv and return its URI (provisional).
Its leading underscore keeps a part-* glob, and a dataset reader that opens the whole directory, from taking
it for a data file.
write_occurrence_buckets
async
¶
write_occurrence_buckets(
results: AsyncIterable[MaterializedVariant],
layout: WideExportLayout,
occurrence_parts: Sequence[str],
destination_uri: str,
*,
storage: Storage,
bucket_size: int,
) -> list[str]
LEFT JOIN results onto the ingest occurrence table and write one Parquet part per ordinal bucket.
This is the portable counterpart of BigQueryScoreStore.export_occurrence_buckets, producing the same
altar-augmentation-buckets-v1 directory altar.variants.export.augmented_vcf.write_augmented_vcf reads:
<destination_uri>/bucket=<n>/part-000000000000.parquet holds every occurrence whose record_ordinal
divides to bucket n by bucket_size, in (record_ordinal, alt_index) order, as the occurrence columns
plus result_variant_id and the wide layout's columns (bucket_schema). An occurrence whose
canonical_variant_id has a result row gets that row; one with no result, or no canonical id at all, has
NULL in every result column, which the augmented-VCF writer reports as NO_RESULT. Only buckets with an
occurrence are written; their URIs are returned in bucket order.
occurrence_parts are the ingest's occurrence shards (<ingest>/occurrences/part-*.parquet, with
OCCURRENCE_ARROW_SCHEMA), read through storage in the order given, which must be record order: a
shard's ordinals follow the previous shard's, as stream_validate_variants writes them.
Neither the results nor a whole bucket is held. The result rows are first spilled to a SQLite file keyed by
variant id, and each occurrence's row is read back from it. A bucket's joined rows go to a local Parquet file
in row groups of _PARQUET_BATCH_ROWS as the occurrences arrive, and the file is uploaded once the ordinals
move past the bucket. Both live in one temporary directory (under TMPDIR), removed when the export returns
or raises. The warehouse export joins in SQL instead.
write_prioritized_ndjson
async
¶
write_prioritized_ndjson(
results: AsyncIterable[MaterializedVariant],
layout: WideExportLayout,
destination_uri: str,
*,
storage: Storage,
) -> str
Write the prioritized variants among results as gzipped NDJSON under destination_uri and return it.
The one part is <destination_uri>/part-000000000000.json.gz, one prioritized_record per line, so a
loader decodes each row with one json.loads. Rows keep the materialize order.
write_wide_csv
async
¶
write_wide_csv(
results: AsyncIterable[MaterializedVariant],
layout: WideExportLayout,
destination_uri: str,
*,
storage: Storage,
prioritized_only: bool = False,
part: str = DOWNLOAD_PART,
provenance: ProvenanceCallback | None = None,
) -> str
Write results as the gzipped, tab-delimited wide download under destination_uri and return that URI.
The one part is <destination_uri>/<part>, part-000000000000.csv.gz by default, the name a warehouse
export's first shard has, with a header row of layout.columns. altar run passes TSV_PART instead, so its
file is named for what it holds. prioritized_only keeps only the prioritized variants, the prioritized
download. Row order is the materialize order, ascending variant_id.
With provenance, the mapping it returns after the last row is written as indented JSON to
<destination_uri>/_provenance.json.
write_wide_parquet
async
¶
write_wide_parquet(
results: AsyncIterable[MaterializedVariant],
layout: WideExportLayout,
destination_uri: str,
*,
storage: Storage,
prioritized_only: bool = False,
provenance: ProvenanceCallback | None = None,
) -> str
Write results as the wide table in Parquet under destination_uri and return that URI.
The one part is <destination_uri>/part-000000000000.parquet with arrow_schema(layout): repeated and
struct annotations keep their native types, so nothing is flattened to text except json columns.
prioritized_only keeps only the prioritized variants. Rows are written a row group at a time as results
yields them, in the same row groups write_parquet_part writes, so the whole result is never held.
With provenance, the mapping it returns after the last row is stored as compact JSON text in the file's
key-value metadata under PROVENANCE_METADATA_KEY (altar.provenance). The columns are unchanged.
long_score_tables
¶
long_score_tables(
layout: WideExportLayout,
) -> tuple[LongScoreTable, ...]
Group the layout's models into score tables by plugin ID and result schema version (provisional).
Tables come in the order of their first model, and each lists its models in layout order. Raises
SchemaCollisionError when a score field is named variant_id, model_id, score_lineage_id or
prioritized, the key columns of every score table.
write_long_export
async
¶
write_long_export(
results: AsyncIterable[MaterializedVariant],
layout: WideExportLayout,
destination_uri: str,
*,
storage: Storage,
export_format: LongExportFormat = "parquet",
prioritized_only: bool = False,
model_info: Mapping[str, LongModelInfo] | None = None,
provenance: ProvenanceCallback | None = None,
) -> dict[str, str]
Write results as the long export under destination_uri and return each table's URI (provisional).
The tables are models, variants and one scores/<table> per result schema (see the module docstring and
long_score_tables), each a directory holding one part, part-000000000000.parquet or, with
export_format="tsv", part-000000000000.tsv.gz with a header row. Parquet keeps the declared types; TSV
writes repeated, struct and json values as JSON text, as the wide TSV does. prioritized_only keeps only the
prioritized variants, and their score rows. model_info adds, by model key, what the layout does not know to
the models table. With provenance, the mapping it returns after the last row is written as indented JSON to
<destination_uri>/_provenance.json.
The returned mapping is keyed by each table's path relative to destination_uri: models, variants, then
scores/<table> in long_score_tables order. Rows keep the materialize order, ascending variant_id, and a
score table's rows for one variant follow the layout's model order. A layout with display-rider columns is
refused, since a rider is computed only by a warehouse query.
annotation_prioritizers_from
¶
annotation_prioritizers_from(
sources: Iterable[Any],
) -> list[AnnotationPrioritizer]
Build an AnnotationPrioritizer for each source that can prioritize a variant.
A source can prioritize when its prioritize_predicate() returns a predicate rather than None;
sources that return None only annotate and are skipped. Each source is used by duck typing — its
.name and .prioritize_predicate() — so AnnotationSource does not have to be imported here.
materialize_variant
¶
materialize_variant(
variant_id: str,
model_scores: Sequence[ModelScore],
annotation: VariantRecord,
annotation_prioritizers: Sequence[
AnnotationPrioritizer
] = (),
) -> MaterializedVariant
Combine one variant's per-model scores and annotations into a MaterializedVariant.
The steps:
- Per-model prioritization: evaluate each model plugin's predicate over the variant's scores merged
with its annotations. Keep the three-valued result (
True,False, orNone) so that a missing input yieldsNonerather than a forcedFalse. model_scores: oneto_model_scoreentry per model, with that model's three-valuedprioritizedincluded.to_model_scoreturns aNoneintoFalse.prioritized_model_ids: the ids of the models whose entry is prioritized.- Model-less prioritizers: evaluate each
AnnotationPrioritizer's predicate over the variant's annotations. A missing column makes the predicateNone, so it does not fire. The ones that do fire are collected inprioritized_source_names. prioritized: true when any model or any prioritizing source isTrue, otherwiseFalse.annotations: the columns ofannotationthat have a value (seepresent_annotations).
The most-active cell type and the headline sort magnitude are not computed here.
bucket_export_layout
¶
bucket_export_layout(
models: Sequence[ScoredModel],
annotation_sources: Sequence[AnnotationSource],
*,
rider: DisplayRider | None = None,
score_lineages: bool = False,
) -> WideExportLayout
Build the wide layout of a result-bucket export: wide_layout with the rider's typed columns (provisional).
Pass only the sources that apply to the models (applicable_annotation_sources). The rider's column types
come from rider_column_dtypes.
bucket_rows_for
¶
bucket_rows_for(
layout: WideExportLayout,
*,
bucket_rows: int | None = None,
bucket_bytes: int | None = None,
) -> int
Return the variants per bucket: bucket_rows, or bucket_bytes (default 128 MiB) over the estimate.
genomic_sort_key
¶
Return the key that orders canonical variant ids in genomic order (provisional).
The key is the chromosome's rank (chromosome_rank), its name, the position, then the reference and alternate
alleles, names and alleles compared as bytes. Distinct canonical ids get distinct keys. Raises
VariantIdentityError for a non-canonical id.
plan_buckets
¶
plan_buckets(
row_count: int, bucket_rows: int
) -> tuple[ResultBucketDescriptor, ...]
Cut row_count ordinals into consecutive buckets of bucket_rows, the last one possibly shorter.
result_bucket_schema
¶
result_bucket_schema(layout: WideExportLayout) -> Schema
Return the Arrow schema of a result bucket for layout (provisional; see the module docstring).
get_score_store_registry
¶
get_score_store_registry() -> PluginRegistry[
type[ScoreStore]
]
Return the registry of score-store plugins registered under altar.score_stores.
safe_identifier
¶
Check that name is a valid identifier before it is placed into a SQL statement, and return it.
Score column names come from plugin.score_columns() in code, but a store that builds SQL by hand still
validates the name rather than trusting it. Raises ValueError if the name is not a plain identifier.
Used by the stores that build SQL as strings (sqlite and BigQuery).