Skip to content

Results and storage

altar.results defines score, detail and artifact stores, and assembles stored scores and annotations into variant results. Store implementations and result readers import from it. Store and assemble results shows how to use it.

Stable, except a few BigQuery export helpers, the list of a wide table's columns and the long export, which are provisional. Provisional names lists them.

results

Stable score-store, schema, and materialized-result API.

Result-backend authors and consumers should import these names from :mod:altar.results. Backend modules under altar.plugins.data_lake remain implementation details even when their classes are re-exported here.

The names a host application uses to subclass or configure the BigQuery store are listed in __provisional__: VariantIdentity, DisplayRider, RiderColumn, export_col_alias and safe_identifier. They are expected to change. So are the wide export's column manifest (wide_column_manifest, ColumnManifestEntry, column_manifest_tsv, write_column_manifest, COLUMN_MANIFEST_FILE) and the long export (write_long_export, long_score_tables, LongScoreTable, LongModelInfo), also provisional.

The result-bucket export is provisional too: the contract a store hands its export over in (ResultBuckets, ResultBucket, ResultBucketDescriptor, RESULT_BUCKET_SCHEMA_VERSION, result_bucket_schema, genomic_sort_key, and bucket_export_layout, bucket_rows_for and plan_buckets, with which a store that produces its own buckets cuts them), the writers that render it into any storage (render_wide_tsv, render_wide_parquet, render_prioritized_ndjson, render_long_export, render_occurrence_buckets), BigQuery's staging setting (ExportStaging), the cost guard's refusal (ExportScanLimitExceeded) and the refusal of an export that repeats a variant (ExportDuplicateRowsError).

LINEAGE_CATALOG_COLUMNS module-attribute

LINEAGE_CATALOG_COLUMNS: Mapping[str, str] = (
    MappingProxyType(
        {
            "score_lineage_id": "STRING",
            "lineage_json": "STRING",
        }
    )
)

The BigQuery columns and types of a lineage catalog, the table named by lineage_table_id, by column name.

Every column is a scalar and required. The catalog holds one row per registered lineage.

LINEAGE_DESCRIPTOR_COLUMNS module-attribute

LINEAGE_DESCRIPTOR_COLUMNS: Mapping[str, str] = (
    MappingProxyType(
        {
            "score_lineage_id": "STRING",
            "plugin_id": "STRING",
            "descriptor_json": "STRING",
            "recorded_at": "TIMESTAMP",
        }
    )
)

The BigQuery columns and types a lineage catalog's descriptor sidecar must have, by column name.

BigQueryScoreStore.lineage_descriptor_table_id names the table. Every column is a scalar, and the table may carry extra nullable columns.

LINEAGE_PROVENANCE_COLUMNS module-attribute

LINEAGE_PROVENANCE_COLUMNS: Mapping[str, str] = (
    MappingProxyType(
        {
            "score_lineage_id": "STRING",
            "provenance_hash": "STRING",
            "run_identity_json": "STRING",
            "recorded_at": "TIMESTAMP",
        }
    )
)

The BigQuery columns and types of a lineage catalog's provenance sidecar, by column name.

BigQueryScoreStore.lineage_provenance_table_id names the table. Every column is a scalar and required. The table holds one row per exact run identity registered under a lineage.

COLUMN_MANIFEST_FILE module-attribute

COLUMN_MANIFEST_FILE = '_columns.tsv'

The file the column manifest of a wide export is written to, beside its part.

RESULT_BUCKET_SCHEMA_VERSION module-attribute

RESULT_BUCKET_SCHEMA_VERSION = 1

The version of the result-bucket schema this module defines; bumped by any change to its columns.

ArtifactKey dataclass

ArtifactKey(
    job_id: str,
    scoring_batch: int | None = None,
    shard: tuple[tuple[str, str], ...] = (),
    variant_id: str | None = None,
)

The identity of the task that produced one artifact file, and of the variant a per-variant file is for.

job_id and scoring_batch are the request's; shard holds any further identity labels the plan's task carried (fold, num_folds), as sorted pairs, so a plan whose terminal tasks each emit a file keeps them apart. variant_id is set only for a variant-scoped artifact: it names the one variant of the task's batch the file is for, so the files one task writes per variant keep distinct keys. The key is provenance: it says which task wrote a file, while the record's variant_ids say what it covers. Records are content-addressed, so a retried batch that produces different bytes under the same key adds a record rather than replacing one.

digest property

digest: str

Return the canonical key hash a store may index.

from_task_labels classmethod

from_task_labels(
    job_id: str,
    scoring_batch: int | None,
    labels: Mapping[str, str],
    *,
    variant_id: str | None = None,
) -> ArtifactKey

Build the key of a container task from the request identity and the task's labels.

The labels every plan sets from the request (job_id, model_id, kind, scoring_batch, scoring_batch_count) are already in the key or are not identity, so only the remaining labels become the shard identity. variant_id names the variant of a variant-scoped file.

ArtifactRecord dataclass

ArtifactRecord(
    model_id: str,
    artifact_name: str,
    lineage_id: str,
    key: ArtifactKey,
    variant_ids: tuple[str, ...],
    media_type: str,
    digest: str,
    byte_size: int,
    uri: str,
)

One stored artifact file: what produced it, what it covers, and where its bytes are.

digest is the sha256: content identity of the bytes and byte_size their length. uri is the location the store put the bytes at, in the vocabulary of its storage; :meth:ArtifactStore.locate turns a record into a readable location for a consumer. A record whose key names a variant covers exactly that variant.

ArtifactStore

Bases: ABC

Persistence axis for binding-declared opaque result files.

Every read takes accepted_lineage_ids, the lineages whose records count, in preference order (the current run's lineage first, as :meth:~altar.results.ScoreReusePolicy.ordered_ids orders them). The empty default places no lineage restriction. Writes name their lineage explicitly: the scoring engine passes the lineage its primary rows will carry.

close

close() -> None

Release the connections and clients this store opened itself.

The default does nothing. A store that opens a connection or creates a client overrides it, and a second call must do nothing. A storage adapter or client passed to the constructor stays open: whoever created it closes it. altar.run closes the artifact store it builds from a run spec when the run ends.

ensure_artifact abstractmethod async

ensure_artifact(
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    schema: ArtifactSchema,
    lineage_id: str | None = None,
) -> None

Create or validate one lineage's artifact without silently changing its schema or plugin identity.

lineage_id is the score lineage of the producing run, the ID its primary rows are stored under; it defaults to the lineage :meth:~altar.results.ScoreLineage.from_run generates for plugin_identity.

put abstractmethod async

put(
    model_id: str,
    artifact_name: str,
    *,
    lineage_id: str,
    key: ArtifactKey,
    variant_ids: Sequence[str],
    source_uri: str,
    storage: Storage,
    digest: str,
) -> ArtifactRecord

Copy the file at source_uri (readable through storage) into the store and index it.

digest is the sha256: identity of the bytes, computed by the caller. The artifact must have been declared with :meth:ensure_artifact under lineage_id first, and key and variant_ids must fit its scope (:func:require_artifact_coverage). A put of bytes the lineage already holds under the same key is idempotent and returns the existing record. The same bytes under another key are another record that may share the stored bytes, so two variants whose per-variant files happen to be identical are each covered.

get_missing_variants abstractmethod async

get_missing_variants(
    model_id: str,
    artifact_name: str,
    variant_ids: Sequence[str],
    accepted_lineage_ids: Sequence[str] = (),
) -> set[str]

Return requested variants no record of this model's artifact lists under an accepted lineage.

read_records abstractmethod async

read_records(
    model_id: str,
    artifact_name: str,
    *,
    variant_ids: Sequence[str] | None = None,
    accepted_lineage_ids: Sequence[str] = (),
) -> AsyncIterator[ArtifactRecord]

Stream stored records, optionally those covering any of variant_ids, under accepted lineages.

locate abstractmethod async

locate(record: ArtifactRecord) -> str

Return a location a consumer can read record's bytes from through the store's storage.

describe_artifact abstractmethod async

describe_artifact(
    model_id: str, artifact_name: str, *, lineage_id: str
) -> StoredArtifact | None

Return durable schema/plugin provenance of one lineage's artifact, or None when it is absent.

StoredArtifact dataclass

StoredArtifact(
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    schema: ArtifactSchema,
    lineage_id: str,
)

Persisted scientific and runtime identity for one model's named artifact under one lineage.

lineage_id is the score lineage of the runs that produced the files: the ID their primary rows carry. plugin_identity is one representative exact identity of that lineage.

supplies

supplies(schema: ArtifactSchema) -> bool

Whether this artifact's files can stand in for schema when its lineage is accepted for reuse.

Accepting a lineage is the caller's scientific decision about the producer; this is the structural half: the same named artifact, at the same declared version, with the same media type and scope. Descriptions do not matter.

BigQueryScoreStore

BigQueryScoreStore(
    client: Any,
    table_id: str,
    *,
    identity: VariantIdentity | None = None,
    lineage_table_id: str | None = None,
    primary_stage_rows: int = 1000000,
    export_staging: ExportStaging | None = None,
    export_maximum_bytes_billed: int | None = None,
    export_staging_max_jobs: int = DEFAULT_STAGING_MAX_JOBS,
    read_client: Any = None,
)

Bases: ScoreStore

BigQuery-backed score store. One table (project.dataset.table) keyed on variant_id and model_id, holding the locus identity columns (VariantIdentity) and one column per plugin score column (added by ensure_schema), clustered for locus pruning.

Build a store over the score table table_id, querying it with client.

client is a google.cloud.bigquery.Client or a compatible fake. table_id is "project.dataset.table". identity is the locus schema, defaulting to VariantIdentity. The client is injected so this module needs no google-cloud-bigquery at import. primary_stage_rows is the size of each staging load job a primary_write unit runs (see primary_write); it bounds the rows a unit buffers in memory and sets how many load jobs an ingest costs against BigQuery's per-table daily quota.

The export settings shape result_buckets (see altar.plugins.data_lake.bigquery_buckets): export_staging is a GCS location the buckets are staged through with EXPORT DATA, separate from where the export's files go; without it they are read with the Storage Read API, through read_client, a google.cloud.bigquery_storage_v1.BigQueryReadClient (or a compatible fake). When it is omitted, the store asks client for one with bigquery_read_client() if it has that method, or else builds one with client's credentials, read from its _credentials attribute, which a wrapper or proxy around a real client passes through. Pass read_client for a client that exposes neither. export_maximum_bytes_billed refuses an export whose dry-run scan estimate is larger, and caps every export job's billing; there is no limit by default. export_staging_max_jobs caps the EXPORT DATA jobs one staged export runs.

client property writable

client: Any

Return the BigQuery client this store runs its queries and load jobs with.

It is read-only. Assigning the client the store already holds is accepted; any other value raises AttributeError.

table_id property writable

table_id: str

Return the score table, as "project.dataset.table".

It is read-only. Assigning the ID the store already holds is accepted; any other value raises AttributeError.

lineage_provenance_table_id property

lineage_provenance_table_id: str | None

Return the lineage catalog's provenance sidecar, "<lineage_table_id>__provenance", or None.

It is None when the store was built without lineage_table_id.

lineage_descriptor_table_id property

lineage_descriptor_table_id: str | None

Return the lineage catalog's descriptor sidecar, "<lineage_table_id>__descriptors", or None.

It is None when the store was built without lineage_table_id. LINEAGE_DESCRIPTOR_COLUMNS lists the columns the table must have. The store never creates it except through ensure_descriptor_table.

connect classmethod

connect(
    table_id: str,
    *,
    credentials: Any = None,
    lineage_table_id: str | None = None,
    identity: VariantIdentity | None = None,
    primary_stage_rows: int = 1000000,
    export_staging: ExportStaging | None = None,
    export_maximum_bytes_billed: int | None = None,
    export_staging_max_jobs: int = DEFAULT_STAGING_MAX_JOBS,
) -> BigQueryScoreStore

Build a store against a real BigQuery client. Requires the altar[bigquery] extra.

The store owns the client it builds here, and close releases it. The other arguments are the constructor's.

from_options classmethod

from_options(
    *,
    table_id: str,
    connection: Connection,
    lineage_table_id: str | None = None,
    primary_stage_rows: int = 1000000,
    export_staging_uri: str | None = None,
    export_maximum_bytes_billed: int | None = None,
    export_staging_max_jobs: int = DEFAULT_STAGING_MAX_JOBS,
) -> BigQueryScoreStore

Build a store from a run spec's [store] options, with a client from connection.bigquery_client().

table_id, lineage_table_id, primary_stage_rows, export_maximum_bytes_billed and export_staging_max_jobs are the constructor's. export_staging_uri, a gs:// prefix, stages exports there, read back through an FsspecStorage on the same connection. The run driver passes the section's named connection; a spec names the connection and never carries a client or credential. The store owns the client it builds here, and close releases it. The connection only says how to reach BigQuery: it never enters a score's identity, so reading the same table through another connection reuses every stored score.

close

close() -> None

Close the BigQuery clients connect, from_options or an export built; a second call does nothing.

A client passed to the constructor is left open.

result_buckets

result_buckets(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    rider: DisplayRider | None = None,
    bucket_rows: int | None = None,
    bucket_bytes: int | None = None,
    variants_relation: str | None = None,
    maximum_bytes_billed: int | None = None,
) -> AbstractAsyncContextManager[ResultBuckets]

Produce the export as result buckets from one canonical table (see ScoreStore.result_buckets).

One CREATE TABLE … AS reads the score table and the annotation sources once into a per-variant table in the score table's dataset, and the buckets are read from it, through export_staging when it is set and the Storage Read API otherwise (altar.plugins.data_lake.bigquery_buckets). The table and any staged shards are removed when the block exits. variants_relation scopes a warehouse-scale job as in materialize; more than 50,000 inline variant_ids are loaded into a scratch table. A dry run estimates the scan first: over maximum_bytes_billed (default: the store's export_maximum_bytes_billed) it raises ExportScanLimitExceeded before any export job runs. Every annotation source must join in SQL. Without models, the portable path annotates the variants.

ensure_schema async

ensure_schema(columns: list[ScoreColumn]) -> None

Create the score table, or add the score columns and the identity column it lacks.

The table's metadata is read first. A column whose physical type differs from the requested one raises SchemaCollisionError. When the table already has every column, no DDL statement runs. BigQuery counts each DDL statement toward the table's daily operation limit, even one that changes nothing. A client that cannot look up table metadata always runs the CREATE TABLE IF NOT EXISTS and ADD COLUMN IF NOT EXISTS statements. With a lineage catalog, the catalog and its provenance sidecar are created the same way, only when missing.

add_scores_frame

add_scores_frame(
    frame: Any,
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    columns: list[ScoreColumn],
    lineage_descriptor: LineageDescriptor | None = None,
) -> InsertResult

Ingest scores from a pandas DataFrame the caller already holds.

This is the columnar path for a caller that already has the scores as a frame — for example a scoring pipeline reading its result TSV in pd.read_csv chunks. It skips add_scores' round-trip of rows to per-row dicts to from_records. frame carries the decomposed locus columns (VariantIdentity: chr/pos/ref/alt) plus one column per score in columns. variant_id is derived here from the locus (chr:pos:ref:alt), and genome (the run's genome_build), model_id, and model_name are stamped on, so the caller supplies only loci and scores. The locus must already be canonical (chr1, not 1; upper-case alleles); a non-canonical one raises VariantIdentityError before anything loads. A missing score is NaN or None in the frame and loads as NULL; an infinite score raises ValueError.

This method is synchronous and blocking, because the columnar bulk load is a blocking BigQuery job, so it is not part of the async ScoreStore interface. A caller on an event loop should offload it to a worker thread (e.g. asyncio.to_thread) so a multi-minute load doesn't stall the loop. The rows-based add_scores is the interface contract; this is a columnar fast path warehouse-scale callers can opt into. With compact lineages, a lineage_descriptor is verified and recorded before the lineage is registered.

stage_scores_frame

stage_scores_frame(
    frame: Any,
    *,
    staging_table: str,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    columns: list[ScoreColumn],
    replace: bool = False,
) -> int

Load one result chunk into a job-specific staging table, not the canonical score table.

The first chunk uses replace=True; later chunks append. The staging table is given a seven-day expiry immediately after its first load, so an interrupted completion cannot leak scratch tables. Call :meth:promote_staged_scores after the complete file is staged, then :meth:drop_staged_scores in a finally block. :meth:primary_write composes exactly that lifecycle over portable rows with a store-chosen staging table, and is what the scoring engine and :func:~altar.scoring.ingest_scoring_results use.

promote_staged_scores async

promote_staged_scores(
    staging_table: str,
    *,
    model_id: str,
    plugin_identity: PluginRunIdentity,
    columns: list[ScoreColumn],
    lineage_descriptor: LineageDescriptor | None = None,
) -> InsertResult

Append a complete, identity-checked staged result to the canonical table.

BigQuery does not enforce unique keys, so concurrent jobs may append the same score key. Read paths defensively select one physical row per key, and deployments may compact old duplicates periodically. The bounded staging source is deduplicated here as an additional guard against repeated scorer rows: each key keeps one whole staged row, the newest by created_at, with its values as a deterministic tie-break.

Promotion independently checks that the staging table contains exactly the expected model and plugin run identity, then refreshes the canonical identity for that model. The insert is also filtered to the expected pair, so rows appended to the staging table between validation and insertion cannot cross the identity boundary.

Deliberately do not MERGE score keys against the canonical cache: it is an unpartitioned, multi-billion-row table, and a source-correlated join cannot prune its clustered blocks. The one canonical read is clustered by model_id and exists only to enforce run identity; score-key promotion still scales with the staged batch rather than the full cache.

With compact lineages, a lineage_descriptor (verified against plugin_identity first) is recorded in the descriptor sidecar before the lineage is registered; the sidecar must be provisioned.

drop_staged_scores async

drop_staged_scores(staging_table: str) -> None

Best-effort caller-facing primitive for removing a score staging table.

primary_write

primary_write(
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    columns: list[ScoreColumn],
    lineage_descriptor: LineageDescriptor | None = None,
) -> AbstractAsyncContextManager[PrimaryScoreWriter]

Stage every batch of the unit in a private staging table, promote it on commit, and always drop it.

This composes stage_scores_frame's load, promote_staged_scores, and drop_staged_scores, so a host no longer calls them itself. The staging table is chosen here, in the score table's dataset, named after the score table, the model, and a random suffix, so concurrent units never share one; the caller never sees it. Each add assembles its rows as add_scores does into a columnar buffer and returns the rows it accepted as added. The buffer is loaded into the staging table in load jobs of exactly primary_stage_rows rows (a constructor option) as it fills, so an ingest of N rows runs ceil(N / primary_stage_rows) load jobs rather than one per engine batch, and the buffer never holds a full load's worth of rows between adds; the first load replaces the table and sets its seven-day expiry. A clean exit loads what remains in the buffer, then promotes the staged rows, which validates the staged model and run identity, rechecks the canonical table's run identity, and (with compact lineages) registers the lineage and the run's provenance, as add_scores does; a unit that accepted no rows loads and promotes nothing. An exception aborts the unit: the buffer is discarded unloaded and nothing is promoted, so none of its rows and none of its lineage provenance reach the canonical tables. Either way the staging table, if a load reached it, is dropped; a failed drop is logged rather than raised (it would otherwise mask the unit's own error), and the expiry removes the table later.

The unit runs ensure_schema(columns) when it opens unless this store already ensured those columns, since promotion inserts into them by name. Without compact lineages the canonical run identity is checked when the unit opens, so an incompatible run fails before anything is staged, and again at promotion. The promotion is one INSERT … SELECT statement, so a commit is all-or-nothing in BigQuery too; like every write to this append-only table it does not dedup against existing keys.

A lineage_descriptor is verified when the unit opens and, with compact lineages, the provisioned descriptor sidecar is checked then too, so a missing sidecar fails before anything is staged; promotion records the descriptor.

get_unscored_variants async

get_unscored_variants(
    variant_ids: Sequence[str],
    model_id: str,
    *,
    variants_relation: str | None = None,
    chromosomes: Sequence[str] | None = None,
) -> list[str]

Return the variant_ids not yet scored by model_id, so a scoring pipeline can skip cache hits.

By default the ids are bound inline as @variant_ids. Pass variants_relation (a backtick-quoted temp-table ref carrying variant_id) to prune at warehouse scale as materialize does, for a variant set too large to bind as an array param. On the relation path the full input lives only in the temp table, so the unscored rows are read straight off a LEFT JOIN … IS NULL dedup and variant_ids may be empty, rather than diffed against an in-memory list. chromosomes then prunes the scores table's chr cluster key, as in export_unscored; it is ignored on the inline path, whose small array binds directly.

Both paths filter on the model's genome build, read from its stored run identity. This store keeps one scientifically compatible identity per model_id, and that identity includes the genome build, so the filter only prunes the genome cluster key and never hides a compatible score. A model with no stored identity has no scores, and the query then runs without a genome filter.

get_missing_scores async

get_missing_scores(
    variant_ids: Sequence[str],
    model_id: str,
    *,
    plugin_identity: PluginRunIdentity,
    columns: Sequence[ScoreColumn],
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    variants_relation: str | None = None,
    chromosomes: Sequence[str] | None = None,
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
) -> list[str]

Filter only candidates, using explicit input genome and the resolved lineages.

variants_relation is the warehouse-scale equivalent of the inline ID list. For large miss sets, use export_unscored to keep them in BigQuery. lineage_descriptor and equivalence resolve as in resolve_lineages.

prepare_unscored_table async

prepare_unscored_table(
    *,
    dest_table: str,
    variants_relation: str,
    model_id: str,
    plugin_identity: PluginRunIdentity,
    columns: Sequence[ScoreColumn],
    max_variants_per_batch: int,
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    chromosomes: Sequence[str] | None = None,
    expiration_timestamp: str = "TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)",
    variant_eligibility: VariantEligibility | None = None,
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
    detail_store: BigQueryDetailStore | None = None,
    detail_schemas: Sequence[DetailSchema] = (),
    artifact_schemas: Sequence[ArtifactSchema] = (),
    check_representation: bool = False,
    reference_fasta: str | PathLike[str] | None = None,
    ineligible_table: str | None = None,
) -> int

Write every cache miss of the candidates in variants_relation to dest_table, numbered into batches.

This is the warehouse form of prepare_scoring_batches, and it sets aside and counts as a miss the same variants. Returns the number of misses written. dest_table holds the locus columns, variant_id and a zero-based scoring_batch, and expires at expiration_timestamp. Export each batch with export_prepared_batch, which reads only this table. The caller owns the candidate relation, the destination and the scoring requests. The candidate relation holds the loci and their variant_id, for the request's genome build.

A candidate is a miss unless an accepted lineage holds its primary row and, for each table in detail_schemas, detail_store holds at least one row for it under a lineage whose table supplies that schema. A detail table whose row_count_field is 0 in the cached primary row needs no row. Pass the plugin's manifest.detail_schemas and the BigQueryDetailStore scoring writes to. The accepted lineages are resolved as in resolve_lineages, with lineage_descriptor and equivalence; a host that froze a resolution passes ScoreReusePolicy.exactly(...) instead. This query cannot check artifact coverage, so a request whose output selection names artifacts (artifact_schemas, from the manifest's selected_artifact_schemas) raises CapabilityNotSupported.

These candidates are set aside rather than prepared, each with the first reason that applies:

  • a variant_id that is not canonical, or does not spell the locus in the other columns, is not_normalized;
  • variant_eligibility, the model's manifest.variant_eligibility, excludes a candidate as unsupported_variant_class or allele_too_long;
  • with check_representation, alleles that are not in minimal representation are not_normalized.

By default no representation check runs, so a canonical candidate whose alleles are not minimal is prepared, as before these checks existed. Pass check_representation=True to set it aside, as prepare_scoring_batches does. These checks run in SQL, so no locus is read to the host. The misses are written to a scratch table in the score table's dataset, and dest_table numbers them into batches from there.

reference_fasta, an indexed FASTA of the run's genome build, turns the representation check on and checks it against the reference. The misses are read back and checked as preparation checks them (representation_check: contig_missing, reference_mismatch, then not_normalized against left alignment), and the rejected loci are left out of dest_table. That read holds at most one result page, 10,000 loci and 100,000 rejected loci in memory at a time.

Pass ineligible_table, written like dest_table, to keep the set-aside candidates. It is replaced with one row per candidate in the columns of preparation's report (INELIGIBLE_VARIANT_COLUMNS), and expires at expiration_timestamp. Without it, the counts are logged as a warning.

count_unscored async

count_unscored(
    *,
    variants_relation: str,
    model_id: str,
    plugin_identity: PluginRunIdentity,
    columns: Sequence[ScoreColumn],
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    chromosomes: Sequence[str] | None = None,
    variant_eligibility: VariantEligibility | None = None,
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
    detail_store: BigQueryDetailStore | None = None,
    detail_schemas: Sequence[DetailSchema] = (),
    artifact_schemas: Sequence[ArtifactSchema] = (),
    check_representation: bool = False,
) -> int

Count the cache misses prepare_unscored_table would find, in one query that writes nothing.

The arguments mean what they mean for prepare_unscored_table, and the count is the one it returns without a reference_fasta. The reference check reads every miss on the host, so this count is taken before it, and a preparation with a reference can set aside more. The lineage catalog is read and never written. This method is provisional.

export_prepared_batch async

export_prepared_batch(
    *,
    prepared_relation: str,
    scoring_batch: int,
    destination_uri: str,
) -> int

Export one previously prepared miss batch without rescanning canonical scores.

The prepared table holds only the loci prepare_unscored_table kept after its eligibility checks.

export_unscored async

export_unscored(
    *,
    model_id: str,
    destination_uri: str,
    variants_relation: str,
    chromosomes: Sequence[str] | None = None,
    scoring_batch: int | None = None,
    scoring_batch_count: int | None = None,
    score_cache_cutoff: datetime | None = None,
    scoring_batch_size: int | None = None,
    plugin_identity: PluginRunIdentity | None = None,
    columns: Sequence[ScoreColumn] = (),
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    variant_eligibility: VariantEligibility | None = None,
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
) -> int

Export the loci of variants in variants_relation not yet scored by model_id, and return the count exported. Writes with EXPORT DATA to destination_uri, an already-resolved gs://…-*.tsv wildcard prefix, staging each model's unscored-variant file. This is a sibling of export_download and export_prioritized; it differs only in the predicate (LEFT JOIN scores … WHERE s.variant_id IS NULL) and a flat headerless locus projection.

variants_relation (a backtick-quoted temp-table ref) is required, and unlike materialize's relation it must carry the decomposed locus columns (VariantIdentity: chr/pos/ref/alt), because the export projects them; the caller's temp table must produce exactly those columns. chromosomes prunes the scores table's chr cluster key; omit it for a correct but unpruned scan. The genome filter is plugin_identity.genome_build when a run identity is given, otherwise the model's stored genome build.

First a parameterized COUNT(*) runs, returning 0 and skipping the export when nothing is unscored. Then the EXPORT DATA OPTIONS(… CSV, tab, header=false) AS dedup SELECT runs. EXPORT DATA forbids query params, so genome, model_id, and chromosomes are inlined into the export SQL (they are controlled internal values) while the count binds them as params.

Pass the model's manifest.variant_eligibility as variant_eligibility so the candidate relation is restricted to variants the model can score before any miss is counted, batched, or exported. Without it an SNV-only model's export includes indels, and run_scoring then rejects their result rows. The manifest's rules are all this export checks; export_unscored_checked also applies preparation's reference and normalization checks to the exported loci and reports the ones it sets aside.

Resolving the gs:// URI, creating the temp table, and composing the resulting shards stay the caller's job; the store's job ends at issuing the export. This does not run on the emulator: the EXPORT DATA GCS sink hangs the job poll, as in export_download, so the generated SQL is checked with a fake client for shape plus an opt-in smoke test against real BigQuery. In compact-lineage mode, lineage_descriptor and equivalence resolve the accepted lineages as in resolve_lineages.

export_unscored_checked async

export_unscored_checked(
    *,
    storage: Storage,
    ineligible_report_uri: str,
    reference_fasta: str | PathLike[str] | None = None,
    model_id: str,
    destination_uri: str,
    variants_relation: str,
    chromosomes: Sequence[str] | None = None,
    scoring_batch: int | None = None,
    scoring_batch_count: int | None = None,
    score_cache_cutoff: datetime | None = None,
    scoring_batch_size: int | None = None,
    plugin_identity: PluginRunIdentity | None = None,
    columns: Sequence[ScoreColumn] = (),
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    variant_eligibility: VariantEligibility | None = None,
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
) -> UnscoredExport

Run export_unscored, then set aside the exported loci the reference rejects or that are not normalized.

This is the host-side check prepare_scoring_batches applies, for the relation-native export: without it, one locus with an unknown contig or a REF that disagrees with the FASTA fails a whole scoring batch after it was dispatched. The export arguments mean what they mean for export_unscored (pass the manifest's variant_eligibility there, as before; it filters in SQL). Each exported file is then read back through storage and checked with the same rules and reasons as preparation (altar.plugins.model.preparation.representation_check): with reference_fasta, the indexed FASTA of the run's genome build, contig_missing, reference_mismatch, then not_normalized against left alignment; without one, only not_normalized against minimal representation. A file that held an ineligible locus is rewritten in place with only its eligible loci, so the batch files a host dispatches hold exactly the eligible loci. The removed loci are written to ineligible_report_uri as the headered TSV preparation writes (INELIGIBLE_VARIANT_COLUMNS, in file order); when none are removed, a report left there by an earlier run is deleted, so the report always describes this export.

destination_uri must hold exactly one *. BigQuery numbers an export's files sequentially from 000000000000, so the files are found by probing those names until one is missing (Storage has no listing); they are returned as shard_uris. ineligible_report_uri must not match the wildcard, or a later listing would read it as a batch file. Reading back more loci than the export counted means files of an earlier, larger export remain under the prefix, and raises RuntimeError; clear the prefix first. The returned exported counts the loci actually read back, which a concurrent writer can make smaller than export_unscored's count.

materialize async

materialize(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    variants_relation: str | None = None,
) -> AsyncIterator[MaterializedVariant]

Stream one per-variant summary at a time from a single fused query.

The query is one SELECT that LEFT JOINs the scores table to each annotation source's sql_join JoinSpec. It computes each model's prioritize_predicate().to_sql() server-side as a per-row is_prioritized column — the same generated SQL the EXPORT paths filter on — and each prioritizing source's predicate as a boolean column. SQL decides prioritization; the model_scores entries are then shaped by each model's to_model_score in Python. A parity test checks the result matches the pure materialize_variant driver row for row, so a caller can trust the pushed-down query.

Streaming: rows arrive ordered by variant_id, so grouping yields one MaterializedVariant at a time without buffering the whole result, which can span millions of variants. That order is the ScoreStore.materialize contract: one row per distinct requested variant, ascending by variant_id.

Scope: one MaterializedVariant per distinct requested variant, as with the portable driver. The fused query covers every variant in variants_relation when one is given. On the inline path it covers the variants a model scored plus those a fused prioritizing source has a row for (see build_scored_cte), so a coding-only variant (e.g. an AlphaMissense missense row with no ChromBPNet score) comes back with empty model_scores and prioritized driven by the source. The remaining requested variants — ones with only passive annotations, or with no evidence at all — are built in Python with empty model_scores, their annotations read and prioritizing sources evaluated over fetch_annotations, and merged into the sorted stream in variant_id order. Every Python annotation read (these, and the annotate() fallback below) fetches a chunk of _MATERIALIZE_CHUNK_VARIANTS sorted requested ids at a time as the sorted stream reaches them, so only one chunk's annotations per read are held. Sources on this backend join in-SQL (sql_join returns a spec); a source on another backend falls back to annotate(), plus Python evaluation when it prioritizes. A passive column referenced by a model predicate must come from a fused source, since it has to be in the query for the predicate to push down.

Annotations: each result's annotations carries the fused sources' columns, which the query already selects for the predicates, plus the other sources' annotate() values, with the same values and the same absent-when-empty rule as the portable driver. A declared scalar json column that the query returns as JSON text (a STRING such as TO_JSON_STRING(...)) is decoded to its JSON value first. The unscored variants outside the fused query read only the fused passive sources again; a fused prioritizing source has no row for them, and the other sources' values were already read. A relation-scoped call with a source that cannot fuse raises ValueError unless the job's ids are also passed in variant_ids, since annotate() needs the ids.

Scoping: variant_ids are bound inline as @variant_ids. Pass variants_relation (a temp-table ref) instead to scope a warehouse-scale job whose id array would exceed BigQuery's request limit; the SQL then prunes via variant_id IN (SELECT variant_id FROM <relation>) and variant_ids may be empty. The two paths return the same result over the same variant set (checked on the emulator). More than 50,000 distinct variant_ids without a variants_relation are loaded into a scratch table in the score table's dataset with a load job, and the query is scoped by that table. The table is dropped once the query has run, and it expires after a day if the drop fails.

export_download async

export_download(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    destination_uri: str,
    rider: DisplayRider | None = None,
    variants_relation: str | None = None,
    prioritized_only: bool = False,
    header: bool = True,
    scored_relation: str | None = None,
    anno_relation: str | None = None,
    storage: Storage | None = None,
) -> str

Export variant_ids as a flat wide TSV to destination_uri (a gs:// prefix) and return it. This is the download branch of the two exports. prioritized_only restricts to prioritized variants (the same rollup as export_prioritized, but in the wide-TSV shape) for the prioritized download. header=False writes headerless parts so a caller doing a server-side GCS compose can prepend a single header (see download_header) instead of one per shard. storage is the base signature's writer; the bulk EXPORT DATA job writes to the prefix itself, so it is accepted and unused.

This is a different artifact from materialize: all variants (not prioritized-only), one row per variant, with each model's score columns pivoted into positional columns (MAX(IF(model_id=…))) rather than a nested ARRAY<STRUCT>. It is the human-downloadable TSV, not the DB rows. It covers the same universe as materialize, every requested variant whether or not a model scored it (inline, the bound @variant_ids; see requested_universe_arm). It shares the same join and pushed-down is_prioritized base (the scored CTE) with materialize, then aggregates: ANY_VALUE per annotation column, and prioritized = COALESCE(LOGICAL_OR(is_prioritized), FALSE) OR … over the fused sources. It runs as one EXPORT DATA OPTIONS(... CSV, tab, GZIP) job. BigQuery cannot export nested or repeated data as CSV, so each annotation column declared repeated or struct is written as its TO_JSON_STRING text, with a missing value left empty. Such a column must physically be an ARRAY or STRUCT, not a pre-serialized string. A scalar json column is exported unchanged. The Parquet and JSON exports keep those columns' native types.

Fused sources only: a flat cloud EXPORT is single-backend, so unlike materialize there is no annotate() fallback. A source that cannot sql_join raises ValueError.

rider is an optional config-injected DisplayRider — for example most_active_celltype and most_active_celltype_logfc display columns, the single model-score row picked per variant by an ordering over the scored namespace. It applies to this aggregated export and the DB export, never to materialize.

This does not run on the emulator: EXPORT DATA … OPTIONS(uri='gs://…') hangs the job poll on the emulator's GCS sink, so the generated SQL is checked with a fake client for shape and the actual GCS write is an opt-in smoke test against real BigQuery. variants_relation scopes as in materialize (a temp-table ref for a warehouse-scale job; None for inline @variant_ids).

scored_relation reads a pre-materialized scored table (from materialize_scored_table) instead of rebuilding and rescanning the fused base CTE. When set, the join and pushed-down is_prioritized are not recomputed here; the export reads the already-computed table, so several exports off one base pay the base scan (which can be multi-TB) once instead of once each. variants_relation and variant_ids are then unused, since the table already carries the job's scope.

anno_relation is the small per-variant table from materialize_annotations_table, paired with a scores-only scored_relation from materialize_scores_table. The export reconstructs scored by LEFT-JOINing them, so each export scans the small scores-only and annotations-once tables instead of a base that duplicates every variant's annotation blobs across all its model rows. This cuts the residual cost below the single-table scan-once path.

export_occurrence_buckets async

export_occurrence_buckets(
    *,
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    occurrence_relation: str,
    result_relation: str,
    destination_uri: str,
    max_record_ordinal: int,
    bucket_size: int,
    rider: DisplayRider | None = None,
    scored_relation: str | None = None,
    anno_relation: str | None = None,
    variant_ids: Sequence[str] = (),
    variants_relation: str | None = None,
) -> list[str]

LEFT JOIN canonical results to occurrences and export one bounded Parquet prefix per ordinal bucket.

The results are the wide download's rows (build_export_query) materialized into result_relation for the job's variants, so each bucket row is the occurrence columns, result_variant_id, and the wide layout's columns; altar.plugins.data_lake.export.write_occurrence_buckets writes the same shape from any store. The job scope is variants_relation (a temp table, typically the ingest's canonical ids) or the inline variant_ids, bound as query parameters of the CTAS exactly as export_download binds them; with a pre-materialized scored_relation the table already carries the scope and neither is needed. Without any of the three the CTAS would reference unbound parameters, so it is refused before any query runs.

Relations and the destination are trusted operator-generated identifiers. Bucket numbers and size are validated integers. The method intentionally issues one EXPORT per bucket: BigQuery object names do not encode row ordering, while the bucket boundary gives the streaming writer a bounded sort unit.

download_header

download_header(
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    rider: DisplayRider | None = None,
) -> list[str]

Return the ordered column names of the export_download wide TSV.

This is the single header line to prepend when the export is written headerless (header=False) for a server-side GCS compose. The columns are the shared wide layout (altar.plugins.data_lake.export.wide_layout, the one build_export_query renders its SELECT aliases from; a test locks this): variant_id, prioritized, the rider's display columns, each fused annotation column, one block per model named with a readable model-name prefix, stable model-id digest, and field suffix, then prioritized_model_ids and prioritized_source_names.

export_prioritized async

export_prioritized(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    destination_uri: str,
    rider: DisplayRider | None = None,
    variants_relation: str | None = None,
    scored_relation: str | None = None,
    anno_relation: str | None = None,
    storage: Storage | None = None,
) -> str

Export only the prioritized variants as NDJSON to destination_uri (a gs:// prefix) and return it. This is the database branch of the two exports. Each row emits model_scores as a native ARRAY<STRUCT>, so a loader can copy a shard into a database and decode each row with one json.loads. That database load is the caller's step, not this method. storage is the base signature's writer; the bulk EXPORT DATA job writes to the prefix itself, so it is accepted and unused.

It differs from export_download in two ways. It keeps only prioritized variants, via a server-side HAVING on the same prioritized rollup over the same requested universe (inline, the bound @variant_ids). It emits model_scores as a native ARRAY<STRUCT> (ARRAY_AGG(STRUCT(…))) rather than pivoting the scores into positional columns, so the exported JSON already has the model_scores shape the database column expects. It shares the fused scored CTE (the join plus the pushed-down is_prioritized) with the other two export surfaces, then aggregates per variant into the full MaterializedVariant contract:

  • model_scores — one struct per model that scored the variant, in to_model_score field order (identity, score columns, COALESCE(is_prioritized, FALSE)). A coding-only variant's single model-less row is dropped (IF(model_id IS NULL, NULL, …) IGNORE NULLS), leaving an empty array.
  • prioritized_model_ids — models whose is_prioritized is TRUE. The three-valued None and FALSE are excluded.
  • prioritized_source_names — each fused prioritizing source's name when its flag fired, in declaration order.

Fused sources only: a cloud EXPORT runs on one backend, so unlike materialize there is no annotate() fallback. A source that cannot sql_join raises ValueError. rider is the same optional display rider as export_download, applied to this aggregated export and never to materialize.

This does not run on the emulator: the EXPORT DATA … OPTIONS(uri='gs://…', format='JSON') GCS sink hangs the job poll, so the generated SQL is checked with a fake client for shape and the actual write is an opt-in smoke test against real BigQuery. The inner SELECT (the ARRAY_AGG(STRUCT) aggregation plus the HAVING filter) does run on the emulator and is parity-tested via build_prioritized_query. variants_relation scopes as in materialize: a temp-table ref at warehouse scale, or None for inline @variant_ids. scored_relation reads a pre-materialized scored table instead of rebuilding the base CTE; pairing it with anno_relation reads the scores-only plus annotations-once split (see export_download).

materialize_scored_table async

materialize_scored_table(
    *,
    dest_table: str,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    variants_relation: str | None = None,
    expiration_timestamp: str | None = None,
) -> str

Materialize the fused scored base once into dest_table with a CTAS, and return dest_table. The join, the pushed-down is_prioritized, and the per-source flags are computed in a single CREATE OR REPLACE TABLE … AS pass over the base scores table, which can be multi-TB. The two downloads and the prioritized NDJSON then export off this table via scored_relation, each a small job-scoped scan, instead of rebuilding and rescanning the base CTE. A job that emits N surfaces scans the base once rather than N times.

The written table is exactly SELECT * FROM scored: one row per (variant, model) carrying variant_id, model_id, model_name, the union of score_columns, every fused annotation column, is_prioritized, and one boolean per prioritizing source. That is the full column set the scored_relation exports project on. variants_relation and variant_ids scope the base scan as in materialize, and the table's universe is that complete scope: a requested variant no model scored gets one model-less row, as materialize yields it, whichever way the scope is given. expiration_timestamp is a SQL TIMESTAMP expression (e.g. TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 6 HOUR)) that sets OPTIONS(expiration_timestamp=…) so a crashed run's temp table self-expires; None writes no expiration and the caller drops the table.

There is no ORDER BY here, unlike materialize's streaming read, so the CTAS streams rows to the table without the global sort that pushed the prioritized export past its memory limit. This does not run on the emulator (a CTAS over the GCS-backed base is real-BigQuery only, like EXPORT); the generated SQL is checked with a fake client for shape, and the base CTE it wraps is the same emulator-tested build_scored_cte.

materialize_scores_table async

materialize_scores_table(
    *,
    dest_table: str,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    variants_relation: str | None = None,
    expiration_timestamp: str | None = None,
) -> str

Materialize the scores-only base into dest_table with a CTAS, and return dest_table. This is the scores half of the annotations-once split. It writes one row per (variant, model) carrying only variant_id, model_id, model_name, and the union of score_columns — no annotation columns and no is_prioritized. The is_prioritized predicate reads region_type, a per-variant annotation, so it can't be computed here and is deferred to the export join. This is a straight projection of the base scores table over the job scope, with no annotation joins and no universe UNION, so the written table holds just the numeric score rows even though the multi-TB base is scanned once.

Pair it with materialize_annotations_table, whose per-variant rows span the same job scope, and export via scored_relation=<this> and anno_relation=<that>. variants_relation and variant_ids scope the scan as in materialize. This does not run on the emulator (a CTAS over the GCS-backed base is real-BigQuery only); the generated SQL is checked with a fake client for shape.

materialize_annotations_table async

materialize_annotations_table(
    *,
    dest_table: str,
    annotation_sources: Sequence[AnnotationSource],
    variant_ids: Sequence[str],
    variants_relation: str | None = None,
    expiration_timestamp: str | None = None,
) -> str

Materialize the per-variant annotations-once table into dest_table with a CTAS, and return dest_table. This is the annotations half of the split. It writes one row per universe variant carrying every fused annotation column — the JSON blobs stored once, not duplicated across a variant's model rows — plus one boolean per prioritizing source (see build_annotations_cte). The universe is the complete job scope, variants_relation or the inline variant_ids, so a requested variant with no result keeps its row and this pass reads only each annotation source once. The base scores table is not rescanned.

Pair it with materialize_scores_table and export via scored_relation and anno_relation. This method sees no models, so pass only the sources applicable_annotation_sources keeps for them. variants_relation and variant_ids also prune each fused source's {variant_filter} as in materialize. This does not run on the emulator (a CTAS is real-BigQuery only); the generated SQL is checked with a fake client for shape, and the two-table export join it feeds is emulator-tested via build_export_query and build_prioritized_query with both relations set.

UnscoredExport dataclass

UnscoredExport(
    exported: int,
    eligible: int,
    ineligible: int,
    ineligible_report_uri: str | None,
    shard_uris: tuple[str, ...],
)

What BigQueryScoreStore.export_unscored_checked exported for one scoring batch, and what it set aside.

exported instance-attribute

exported: int

Loci the export wrote: every row read back from its files (0 when nothing was unscored).

eligible instance-attribute

eligible: int

Loci left in the batch files after the ineligible ones were removed.

ineligible instance-attribute

ineligible: int

Loci removed because the reference rejects them or they are not normalized.

ineligible_report_uri instance-attribute

ineligible_report_uri: str | None

The ineligible-variant report, or None when nothing was set aside (a stale report is then deleted).

shard_uris instance-attribute

shard_uris: tuple[str, ...]

The export's files in shard order, each holding only eligible loci; some may be empty.

VariantIdentity dataclass

VariantIdentity(
    locus_columns: tuple[tuple[str, str], ...] = (
        ("chr", "str"),
        ("pos", "int"),
        ("ref", "str"),
        ("alt", "str"),
    ),
    clustering: tuple[str, ...] = (
        "genome",
        "model_id",
        "chr",
        "variant_id",
    ),
)

The locus columns a BigQuery score table carries in addition to the variant_id join key.

These are genome plus the decomposed chr/pos/ref/alt locus, plus the clustering keys, so the store can cluster and prune scans by locus. variant_id is the portable join key the read and materialize code is built on; these columns are the source-of-truth locus and cluster keys the caller populates, and variant_id is chr:pos:ref:alt derived from them. A deployment can override the locus columns or the clustering. There is no default genome: genome is always the scoring run's genome_build.

ExportStaging dataclass

ExportStaging(uri: str, storage: Storage)

Where BigQueryScoreStore stages an export's Parquet shards, and how it reads them back (provisional).

uri is a gs:// prefix (BigQuery's EXPORT DATA writes only to GCS) and storage a Storage that reads and deletes objects under it, such as an FsspecStorage. It is a store setting, separate from the Storage the export's files are written to, which can be anything.

BigQueryDetailStore

BigQueryDetailStore(client: Any, table_id: str)

Bases: DetailStore

Generic BigQuery adapter for every binding-declared :class:DetailSchema.

table_id names the canonical JSON-row table. Its metadata sidecar is named {table_id}__tables and exact compatible provenance is retained in {table_id}__provenance. All schemas are fixed and contain no binding- or model-specific columns. Writes stage one bounded batch and MERGE on (model_id, detail_name, lineage_id, logical_key), making retries idempotent while preserving null-bearing logical keys through their canonical digest.

connect classmethod

connect(
    table_id: str, *, credentials: Any = None
) -> BigQueryDetailStore

Build a store against a real BigQuery client. Requires the altar[bigquery] extra.

The store owns the client it builds here, and close releases it.

from_options classmethod

from_options(
    *, table_id: str, connection: Connection
) -> BigQueryDetailStore

Build a store from a run spec's [detail_store] options, with connection.bigquery_client().

The store owns the client it builds here, and close releases it. The connection never enters any identity, and a spec never carries the client or a credential.

close

close() -> None

Close the BigQuery client connect or from_options built; a second call does nothing.

A client passed to the constructor is left open.

get_missing_variants async

get_missing_variants(
    *,
    model_id: str,
    detail_name: str,
    variant_ids: Sequence[str],
    accepted_lineage_ids: Sequence[str] = (),
) -> set[str]

Return requested variants without rows, reading only the clustered identity columns.

The rows table is unpartitioned and CLUSTER BY model_id, detail_name, lineage_id, variant_id. Each chunk filters by equality on the first two clustering columns and IN UNNEST on the others, so BigQuery can prune storage blocks to the requested model/table, lineages, and variants instead of scanning the table. Only those narrow STRING columns are referenced; row_json is never read, so a wide detail table (thousands of Enformer tracks per variant) adds rows but not payload bytes to the scan. One query runs per 10,000 IDs; :func:~altar.scoring.prepare_scoring_batches already bounds each call by its cache_query_size (default 10,000) and passes only primary cache hits.

DetailRowKey dataclass

DetailRowKey(
    variant_id: str, values: tuple[DetailKeyValue, ...]
)

Backend-neutral logical identity for one repeated result row.

digest property

digest: str

Return the canonical key hash stores may use for NULL-safe uniqueness.

DetailStore

Bases: ABC

Persistence axis for binding-declared named detail tables.

Every read takes accepted_lineage_ids, the lineages whose rows count, in preference order (the current run's lineage first, as :meth:~altar.results.ScoreReusePolicy.ordered_ids orders them). The empty default places no lineage restriction: rows of every lineage stored under the model count.

close

close() -> None

Release the connections and clients this store opened itself.

The default does nothing. A store that opens a connection or creates a client overrides it, and a second call must do nothing. A client passed to the constructor stays open: whoever created it closes it. altar.run closes the stores it builds from a run spec when the run ends.

ensure_table abstractmethod async

ensure_table(
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    schema: DetailSchema,
    lineage_id: str | None = None,
) -> None

Create or validate the table of one lineage without silently changing its schema or plugin identity.

lineage_id is the score lineage of the producing run, the ID its primary rows are stored under; it defaults to the lineage :meth:~altar.results.ScoreLineage.from_run generates for plugin_identity.

get_missing_keys abstractmethod async

get_missing_keys(
    *,
    model_id: str,
    detail_name: str,
    keys: Sequence[DetailRowKey],
    accepted_lineage_ids: Sequence[str] = (),
) -> set[DetailRowKey]

Return requested logical keys not persisted for this model and table under an accepted lineage.

get_missing_variants async

get_missing_variants(
    *,
    model_id: str,
    detail_name: str,
    variant_ids: Sequence[str],
    accepted_lineage_ids: Sequence[str] = (),
) -> set[str]

Return requested parent variants with no persisted row in this model's table under an accepted lineage.

A detail table is one-to-many, so the logical keys a variant will produce are unknown before inference. This parent-level presence query lets cache-miss preparation treat a variant with no stored detail rows as incomplete without enumerating keys. It does not prove that a variant with some rows has all of them; callers combine it with the engine's details-before-primary write order.

The default streams :meth:read_rows and is correct for any adapter. Adapters should override it with an indexed existence query so that wide tables (thousands of tracks per variant) are not read.

add_rows abstractmethod async

add_rows(
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    schema: DetailSchema,
    rows: Sequence[Mapping[str, object]],
    lineage_id: str | None = None,
) -> InsertResult

Persist schema-shaped rows under one lineage and report adapter-side duplicate skips.

lineage_id is resolved as :meth:ensure_table resolves it.

read_rows abstractmethod async

read_rows(
    *,
    model_id: str,
    detail_name: str,
    variant_ids: Sequence[str] | None = None,
    accepted_lineage_ids: Sequence[str] = (),
) -> AsyncIterator[VariantRecord]

Stream stored rows, optionally restricted to parent variants and to accepted lineages.

With accepted lineages, a variant's rows come from the first lineage in accepted_lineage_ids that holds rows for it, as primary reads choose one lineage per variant. Rows of a later lineage are not read for that variant, so two producers' rows never mix. With none, rows of every lineage are read.

describe_table abstractmethod async

describe_table(
    model_id: str, detail_name: str, *, lineage_id: str
) -> StoredDetailTable | None

Return durable schema/plugin provenance of one lineage's table, or None when it is absent.

StoredDetailTable dataclass

StoredDetailTable(
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    schema: DetailSchema,
    lineage_id: str,
)

Persisted scientific and runtime identity for one model's named detail table under one lineage.

lineage_id is the score lineage of the run that produced the table: the ID its primary rows carry. plugin_identity is one representative exact identity of that lineage.

supplies

supplies(schema: DetailSchema) -> bool

Whether this table's rows can stand in for rows of schema when its lineage is accepted for reuse.

Accepting a lineage is the caller's scientific decision about the producer; this is the structural half: the stored table must be the same named table, at the same declared version, with the same row identity and field meanings. Labels and descriptions do not matter.

DisplayRider dataclass

DisplayRider(order_by: str, columns: Sequence[RiderColumn])

Configuration for a variant's display columns.

It picks the single model-score row that sorts first by order_by (over the scored columns) and emits columns from it.

This renders to two SQL steps. The first runs ROW_NUMBER() OVER (PARTITION BY variant_id ORDER BY <order_by>, <model position>), with is_prioritized read as FALSE when a decision is missing; the second keeps rank = 1 and evaluates the named column expressions on it. It uses ROW_NUMBER rather than ARRAY_AGG(… LIMIT 1)[OFFSET(0)] so it runs on the BigQuery emulator.

  • order_by: a SQL ORDER BY fragment that selects which of a variant's rows to emit from — for example "is_prioritized DESC, ABS(logfc) DESC" (prioritized models first, then largest |logFC|).
  • columns: the ordered output columns emitted from the picked row.

RiderColumn dataclass

RiderColumn(
    name: str, expr: str, dtype: ScalarDtype | None = None
)

One output column of a display rider.

name is the emitted column, and expr is a SQL expression evaluated over the picked score row's columns (the score columns plus is_prioritized, model_id, and model_name).

For example, RiderColumn("most_active_celltype", "model_name") and RiderColumn("most_active_celltype_logfc", "logfc") emit the picked model's display name and its logFC. The store keeps model_name among the scored columns, so expr can be a bare column reference rather than a CASE model_id WHEN … THEN name lookup.

dtype is the column's scalar type in typed exports (the result buckets, Parquet, the column manifest). A bare column reference infers it (rider_column_dtypes); any other expression must declare it.

ColumnManifestEntry dataclass

ColumnManifestEntry(
    column: str,
    kind: ColumnKind,
    field: str,
    dtype: str | None,
    repeated: bool = False,
    label: str | None = None,
    source: str | None = None,
    model_id: str | None = None,
    model_name: str | None = None,
    plugin_id: str | None = None,
    plugin_version: str | None = None,
    result_schema_version: str | None = None,
    result_schema_hash: str | None = None,
)

One column of a wide export, as wide_column_manifest describes it (provisional).

  • column is the column's name in the file, and kind what it holds: variant_key (variant_id), prioritization (the variant's prioritized and its two reason lists, or one model's prioritized), display (a warehouse display rider's column), annotation (a source's column), score (a model's score field) or lineage (a model's score_lineage_id).
  • field is the name the value has outside the wide file: a score field's own name, such as log2fc, which models that share a result schema share; an annotation's column name; or the column's name otherwise.
  • dtype is the logical type, float, int, bool, str, json or struct, and repeated says the column is a list of it. A display rider's column has its type when the layout knows it (a result-bucket export) and None otherwise. A TSV holds a repeated, struct or json value as JSON text.
  • label is a score or annotation column's display label.
  • source names an annotation column's source.
  • The model fields are set on a model's columns: model_id (the model key), model_name, plugin_id, plugin_version, and the result_schema_version and result_schema_hash of the plugin's result schema.

to_dict

to_dict() -> dict[str, Any]

Return the entry as JSON-compatible values, keyed in COLUMN_MANIFEST_FIELDS order.

WideExportLayout dataclass

WideExportLayout(
    columns: tuple[str, ...],
    rider_columns: tuple[str, ...],
    annotation_columns: tuple[AnnotationColumn, ...],
    model_blocks: tuple[ModelExportBlock, ...],
    score_lineages: bool,
    annotation_source_names: tuple[str, ...] = (),
    rider_dtypes: tuple[ScalarDtype, ...] = (),
)

The column layout of the wide exports, built by wide_layout.

columns is every column in order. row renders one MaterializedVariant into that layout with logical values: an absent annotation and every field of a model that did not score the variant are None. The writers encode those values per format. annotation_source_names names the source of each of annotation_columns, in the same order. rider_dtypes is the scalar type of each of rider_columns when it is known (a layout built for result buckets knows them), and empty otherwise.

json_text_columns

json_text_columns() -> frozenset[str]

Return the annotation columns a CSV cell holds as JSON text: repeated, struct, and json columns.

DescriptorTableNotProvisioned

DescriptorTableNotProvisioned(message: str, *, table: str)

Bases: CapabilityNotSupported

Raised when a store must record a lineage descriptor but its host has not provisioned the descriptor table.

A store that never creates the table implicitly (BigQuery) raises this instead of a bare CapabilityNotSupported, so a caller can tell a table waiting to be provisioned from a store that keeps no descriptors at all. table names the table to provision, or to create with ensure_descriptor_table().

DescriptorTableNotProvisionedWarning

Bases: UserWarning

Emitted when an engine path cannot record a run's lineage descriptor because its table is not provisioned.

The run proceeds and its scores are written, but the lineage takes part in read-time equivalence only once its descriptor is recorded. Provision the named table, or call ensure_descriptor_table(), and score or prepare the configuration again to record it. prepare_scoring_batches, run_scoring, and ingest_scoring_results emit it at most once per store instance and table.

LongModelInfo dataclass

LongModelInfo(
    lineage_id: str | None = None,
    registry: str | None = None,
    registry_id: str | None = None,
    registry_entry_sha256: str | None = None,
)

What a caller knows about one model beyond the layout, for its row of the models table (provisional).

lineage_id is the lineage the model's run requested; the rows themselves carry the lineage their scores came from in score_lineage_id. registry, registry_id and registry_entry_sha256 name the registry entry the model came from, when it came from one.

LongScoreTable dataclass

LongScoreTable(
    name: str,
    plugin_id: str,
    result_schema_version: str,
    model_ids: tuple[str, ...],
    fields: tuple[ExportField, ...],
)

One score table of the long export: the models whose plugin and result schema version it holds (provisional).

name is the table's directory under scores/. fields are its score columns, each named by its own field name, in the order the models declare them; a field one model of the table lacks is NULL in that model's rows.

path property

path: str

The table's directory, relative to the export: scores/<name>.

AnnotationPrioritizer dataclass

AnnotationPrioritizer(name: str, predicate: Predicate)

A reference source that can prioritize a variant without a model, such as AlphaMissense.

It pairs a name with a Predicate that decides whether the variant is prioritized, evaluated over the variant's annotation columns. Unlike a model, it has no model_id and produces no score entry. When its predicate is true, it only contributes to the overall prioritized flag. annotation_prioritizers_from builds these from the configured sources.

MaterializedVariant dataclass

MaterializedVariant(
    variant_id: str,
    model_scores: list[dict[str, Any]],
    prioritized: bool,
    prioritized_model_ids: list[str],
    prioritized_source_names: list[str],
    plugin_identities: dict[str, dict[str, Any]],
    score_lineages: dict[str, str] = dict(),
    annotations: dict[str, Any] = dict(),
)

The per-variant summary produced by materialize_variant.

It holds only the fields that are the same across every model architecture. The optional display values — the headline score magnitude and the most-active cell type — are decided by each plugin and are not fields here.

annotations maps each declared annotation column that has a value for this variant to that value, merged across every configured source. A column with no value is absent rather than None: a source with no row for the variant, a null cell, and an empty repeated value all leave the column out (BigQuery returns a null array as an empty one, so no store can tell those apart). A variant no source annotated gets {}. Values keep their logical type on every store: a repeated column is a list and a struct column a dict (a list of dicts when repeated), never the JSON text a backend such as SQLite stores them as. Values nested inside a struct or list are kept as the source returned them.

ModelScore dataclass

ModelScore(model: ScoredModel, score: VariantRecord)

One model's raw score for a variant, keyed by the model's score_columns (for example logfc or in_peak).

ScoredModel dataclass

ScoredModel(
    model_id: str,
    model_name: str,
    plugin: ModelResults,
    plugin_identity: PluginRunIdentity,
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    lineage_descriptor: LineageDescriptor | None = None,
)

A model that scored the job's variants, together with its architecture plugin.

plugin is typed as ModelResults, the smaller interface that exposes only the methods for reading results. Materialization never needs the model-building methods (build_*), so it cannot call them by accident. A full ModelPlugin also satisfies this type.

lineage_descriptor class-attribute instance-attribute

lineage_descriptor: LineageDescriptor | None = None

The run's lineage descriptor, so a compact-lineage store reads equivalent lineages before it is recorded.

equivalence property

equivalence: ModelPlugin | None

The plugin whose equivalence rules resolve this model's lineages: plugin, when it is a ModelPlugin.

ExportDuplicateRowsError

ExportDuplicateRowsError(
    row_count: int,
    variant_count: int,
    *,
    where: str = "the export's result table",
)

Bases: ValueError

An export holds more than one row for some variant, where its contract is one row per variant (provisional).

row_count rows hold variant_count distinct variants. BigQueryScoreStore checks its result table right after building it and raises this before any EXPORT DATA job or Read API session, so nothing has been written. ResultBuckets raises it for any store's bucket that repeats a variant, before a writer renders that bucket. The usual cause is an annotation source that returns several rows for one variant, such as a reference table loaded twice or a relation= that does not aggregate per variant. The stores collapse each source to one row per variant, so this means a store's query or a source broke that rule.

ExportScanLimitExceeded

ExportScanLimitExceeded(
    estimated_bytes: int, maximum_bytes_billed: int
)

Bases: RuntimeError

A store refused an export whose estimated scan exceeds the configured maximum_bytes_billed.

Raised before any export job runs, so nothing is billed. estimated_bytes is the store's estimate (for BigQuery a dry run, an upper bound on a clustered table) and maximum_bytes_billed the configured limit. A host catches it to record the export as unavailable rather than fail the job.

ResultBucket dataclass

ResultBucket(
    descriptor: ResultBucketDescriptor, table: Table
)

One bucket's rows (provisional): a table with result_bucket_schema, its rows in no particular order.

sorted_table

sorted_table() -> Table

Return the bucket's rows sorted by variant_ordinal, the order every writer renders them in.

ResultBucketDescriptor dataclass

ResultBucketDescriptor(
    index: int,
    first_ordinal: int,
    last_ordinal: int,
    row_count: int,
    sort_key: tuple[str, ...] = (VARIANT_ORDINAL,),
    schema_version: int = RESULT_BUCKET_SCHEMA_VERSION,
)

One bucket of an export (provisional): its number, the ordinals it covers, and its row count.

first_ordinal and last_ordinal are inclusive, and row_count is last_ordinal - first_ordinal + 1: every variant of the export has exactly one row. sort_key names the column a writer sorts the bucket's rows by.

ResultBuckets

ResultBuckets(
    layout: WideExportLayout,
    descriptors: Sequence[ResultBucketDescriptor],
    load: BucketLoader | None = None,
    *,
    models: Sequence[ScoredModel] = (),
    reader: BucketReader | None = None,
)

The buckets of one export, as a store produces them (provisional).

layout is the export's wide layout (with the rider's columns and types), schema the bucket schema, and descriptors the buckets in genomic order. Iterating yields each ResultBucket in that order, reading one at a time through the producer's loader, which a later iteration calls again; a writer that drops each bucket before asking for the next holds one bucket. Each table is cast to schema, an empty repeated annotation is turned into null, a table that repeats a variant_ordinal raises ExportDuplicateRowsError, and one whose row count or ordinals do not match its descriptor raises ValueError. models are the export's models, in order.

A producer passes load, a loader every iteration shares, or reader, which opens a loader of its own for each iteration and closes it when the iteration ends: a producer whose reading keeps state, such as the local copies of staged files, uses it so that several writers can iterate the same buckets at once.

row_count property

row_count: int

The number of variants in the export, across every bucket.

DescriptorAttachmentFailure dataclass

DescriptorAttachmentFailure(
    identity: PluginRunIdentity,
    descriptor: LineageDescriptor,
    reason: str,
)

One (identity, descriptor) pair ScoreStore.attach_lineage_descriptors did not record, and why.

DescriptorAttachmentReport dataclass

DescriptorAttachmentReport(
    attached: tuple[str, ...] = (),
    unregistered: tuple[str, ...] = (),
    failed: tuple[DescriptorAttachmentFailure, ...] = (),
)

What ScoreStore.attach_lineage_descriptors did, by lineage ID in input order.

attached lineages are registered and now carry their descriptor; unregistered ones had their descriptor recorded but have no catalog entry yet, so they join equivalence once registered; failed pairs did not verify or conflict with a recorded descriptor, and recorded nothing.

Resolution

Bases: tuple[ScoreLineage, ...]

The lineages a read accepts, in preference order: a tuple of ScoreLineage with how each was chosen.

The current lineage comes first, then the registered lineages the binding declares equivalent (by lineage ID), then the policy's accepted lineages. steps holds each lineage's ResolutionStep, and rules_id the plugin's equivalence_rules_id when the equivalence step ran, else None. Freeze it with ScoreReusePolicy.exactly(resolution.lineage_ids).

lineage_ids property

lineage_ids: tuple[str, ...]

The resolved lineage IDs, in preference order.

steps property

steps: tuple[ResolutionStep, ...]

How each lineage was chosen, aligned with the tuple.

rules_id property

rules_id: str | None

The equivalence_rules_id the equivalence step applied, or None when it did not run.

ResolutionStep

Bases: StrEnum

How a lineage entered a Resolution.

ScoreLineage dataclass

ScoreLineage(
    lineage_id: str,
    plugin_id: str,
    columns: tuple[ScoreColumn, ...],
    metadata_json: str = "{}",
    run_identity: PluginRunIdentity | None = None,
)

An immutable catalog entry referenced by a compact ID on score rows.

An imported lineage may cover many models and input genomes: those remain separate row keys. metadata_json holds publisher/operator declarations, including any model-specific settings. Historical imports need no invented Altar run identity. Registering a lineage does not by itself accept it for reuse.

from_run classmethod

from_run(
    identity: PluginRunIdentity,
    columns: Sequence[ScoreColumn],
) -> ScoreLineage

Key the lineage on the run's scientific projection; keep the exact identity for provenance.

The ID hashes PluginRunIdentity.scientific_dict(), so two runs share a lineage exactly when their results are scientifically interchangeable (see same_lineage). Exact provenance (resource locations, the declared configuration serialization, framework versions) never decides reuse and is never repeated on score rows. A lineage store keeps one deterministic representative run identity as the catalog's run_identity and records every distinct exact identity that registers the lineage to write scores; ScoreStore.read_lineage_provenance returns that set.

same_lineage

same_lineage(other: ScoreLineage) -> bool

Whether other declares this lineage, so a catalog holding either accepts the other.

Both must name the same ID, plugin, output columns, and metadata. A generated lineage (one with a run_identity) is defined by its run's PluginRunIdentity.scientific_dict(), so declarations from runs that differ only in exact provenance agree. An imported lineage has no run identity and agrees only with an identical declaration; it never agrees with a generated one.

validate_outputs

validate_outputs(
    plugin_id: str, columns: Sequence[ScoreColumn]
) -> None

Acceptance cannot cross plugins or substitute missing/incompatible outputs.

ScoreReusePolicy dataclass

ScoreReusePolicy(
    accepted_lineage_ids: tuple[str, ...] = (),
    exact: bool = False,
)

Additional acceptable lineages, in preference order after the current run.

This is an explicit scientific decision by the caller, scoped to the model and genome being queried. It is neither inferred from version numbers nor transitive. An empty policy accepts only the current run, plus the lineages its binding declares equivalent when the store is given the plugin and a descriptor. Acceptance changes reads, never the lineage stamped on newly computed scores.

An exact policy, built with :meth:exactly, instead lists every lineage to read, the current one first, and resolves to exactly that list.

exactly classmethod

exactly(lineage_ids: Iterable[str]) -> Self

Read exactly lineage_ids, in that order of preference, such as a frozen Resolution.

The first ID must be the run's current lineage, which need not be registered; every other ID must be. Equivalent lineages are not added. A Resolution passes its lineage_ids.

ordered_ids

ordered_ids(current: ScoreLineage) -> tuple[str, ...]

Return the lineage IDs this policy names for a run whose current lineage is current, current first.

StoredScore dataclass

StoredScore(values: VariantRecord, lineage: ScoreLineage)

A selected score and the lineage it was stored under, including imported provenance.

In compact-lineage mode lineage.run_identity is the lineage's reference identity, one deterministic representative of the runs that wrote under it; a row is not attributed to the exact identity that wrote it. ScoreStore.read_lineage_provenance returns the set of exact identities that wrote under the lineage.

SchemaCollisionError

Bases: ValueError

Raised when independently owned plugin columns cannot be composed safely.

PrimaryScoreWriter

Bases: ABC

The writer one ScoreStore.primary_write unit yields: it accepts the unit's primary rows batch by batch.

Every batch belongs to the model, run identity, and columns the unit was opened with. add takes rows in the shape ScoreStore.add_scores takes and returns an InsertResult for that batch: added counts the rows the unit accepted, skipped the rows the store declined because it already held them. For a store whose unit is atomic, the counted rows become visible only when the unit commits, and none of them do if it aborts.

add abstractmethod async

add(rows: Tabular) -> InsertResult

Stage or write one batch of primary rows, and return its counts.

ScoreStore

Bases: ABC

Base class for a store that writes and reads per-model variant scores.

Stores register under altar.score_stores and are resolved by name. name is the registry key, for example "sqlite" or "bigquery".

A note on the parameter types. At the call sites, rows (and, for a warehouse store, variant_ids) may be pandas objects, but core itself does not depend on pandas, so they are typed as Tabular (a name for Any documented in types); the conformance suite is what actually pins their shape. The read methods return the portable VariantRecord instead. The columns a store persists come from plugin.score_columns(), which is the one place the score-column set is defined.

atomic_primary_write class-attribute

atomic_primary_write: bool = False

Whether primary_write publishes all of a unit's rows on commit and none of them on abort.

lineage_provenance_table_id property

lineage_provenance_table_id: str | None

Return the table that records each exact run identity under a lineage, or None without a catalog.

A host that provisions or checks the lineage tables itself reads the name here. Stores that keep a lineage catalog override this; the base store keeps none.

lineage_descriptor_table_id property

lineage_descriptor_table_id: str | None

Return the table that holds one LineageDescriptor per lineage, or None without a catalog.

Stores that keep a lineage catalog override this; the base store keeps none.

close

close() -> None

Release the connections and clients this store opened itself.

The default does nothing. A store that opens a connection or creates a client overrides it, and a second call must do nothing. A client passed to the constructor stays open: whoever created it closes it. altar.run closes the stores it builds from a run spec when the run ends.

register_score_lineage async

register_score_lineage(
    lineage: ScoreLineage,
    *,
    lineage_descriptor: LineageDescriptor | None = None,
) -> None

Register immutable metadata before importing/backfilling its compact row ID.

This does not rewrite scores or accept the lineage for any request. Re-registering a declaration that ScoreLineage.same_lineage accepts is idempotent and keeps the catalog entry; any other change to an existing ID fails. A generated lineage may pass its lineage_descriptor, which is verified against its run identity and recorded first.

ensure_descriptor_table async

ensure_descriptor_table() -> None

Explicitly create this store's lineage descriptor sidecar if it is missing, and check its columns.

A host may provision the sidecar itself instead; extra nullable columns are accepted. SQLite also creates it on the first descriptor it records; BigQuery never creates it except through this call.

resolve_lineages async

resolve_lineages(
    plugin_identity: PluginRunIdentity,
    columns: Sequence[ScoreColumn],
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    *,
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
) -> Resolution

Return the lineages a read under plugin_identity accepts, in preference order, without reading scores.

The Resolution lists the current lineage, then (when equivalence, the run's ModelPlugin, is given) every registered lineage whose descriptor that plugin's equivalence_projection makes equal to the current one, then reuse_policy's accepted lineages. The current descriptor is lineage_descriptor, verified against the identity, or else the recorded one. Freeze a resolution with ScoreReusePolicy.exactly(resolution.lineage_ids) to read it back unchanged later.

record_lineage_descriptor async

record_lineage_descriptor(
    plugin_identity: PluginRunIdentity,
    descriptor: LineageDescriptor,
) -> None

Verify descriptor against plugin_identity and record it for the run's lineage.

Recording needs no registration, so a host records the descriptor at admission and the lineage joins equivalence when a write registers it. Recording again is a no-op; a different descriptor is refused.

attach_lineage_descriptors async

attach_lineage_descriptors(
    items: Iterable[
        tuple[PluginRunIdentity, LineageDescriptor]
    ],
) -> DescriptorAttachmentReport

Verify and record descriptors for existing lineages, such as every registration a host holds.

Every pair whose descriptor verifies against its identity is recorded, registered or not; the report separates registered lineages from unregistered ones and lists the pairs that failed.

lineages_without_descriptors async

lineages_without_descriptors(
    plugin_id: str,
) -> tuple[str, ...]

Return the registered lineages of plugin_id that have a run identity but no recorded descriptor.

A descriptor backfill is complete when each one is attached or known to be unattachable.

read_lineage_provenance async

read_lineage_provenance(
    lineage_id: str,
) -> tuple[PluginRunIdentity, ...]

Return the set of distinct exact run identities under a lineage, oldest record first.

A generated lineage is keyed on its run's scientific projection, so runs that differ only in exact provenance (resource locations, score-neutral configuration serialization, framework API versions) share it and reuse each other's scores. The catalog keeps one deterministic reference run_identity per lineage, always included here; each run that registers to write scores under the lineage (even if all its rows are skipped as duplicates) or registers its generated lineage explicitly is recorded once. Rows are attributed to the lineage, not to one of these identities. An imported lineage, or an ID this store never registered, has none.

get_missing_scores async

get_missing_scores(
    variant_ids: Sequence[str],
    model_id: str,
    *,
    plugin_identity: PluginRunIdentity,
    columns: Sequence[ScoreColumn],
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
) -> list[str]

Return candidate misses under a model-, genome-, and lineage-scoped policy.

With compact lineages, lineage_descriptor and equivalence resolve as resolve_lineages does.

read_selected_scores async

read_selected_scores(
    variant_ids: Sequence[str],
    model_id: str,
    *,
    plugin_identity: PluginRunIdentity,
    columns: Sequence[ScoreColumn],
    reuse_policy: ScoreReusePolicy = ScoreReusePolicy(),
    lineage_descriptor: LineageDescriptor | None = None,
    equivalence: ModelPlugin | None = None,
) -> dict[str, StoredScore]

Select one whole row per variant, retaining its actual lineage.

Current-run results win; otherwise the first available lineage in resolution order wins (see resolve_lineages, whose lineage_descriptor and equivalence this takes). Output presence is a catalog declaration, not a test for non-NULL values.

required_row_identity

required_row_identity() -> dict[str, str]

Return typed row-identity fields this adapter requires in addition to variant_id.

The common engine preserves only explicitly declared portable identity fields. Persistence-only stores normally need no decomposed locus and inherit the empty mapping. Warehouse stores can require the portable chr/pos/ref/alt locus without teaching a model binding about that backend: the engine derives omitted locus fields from the canonical variant_id and verifies emitted ones. Portable fields keep their portable dtypes; other required fields must be emitted by the codec.

ensure_schema abstractmethod async

ensure_schema(columns: list[ScoreColumn]) -> None

Additively set up the backing store to hold columns plus shared variant identity.

Existing model schemas remain intact. Reusing a column name is valid when its storage definition agrees with the existing one; display-label changes do not alter stored values. Both reference stores take the column set from score_columns().

add_scores abstractmethod async

add_scores(
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    rows: Tabular,
    columns: list[ScoreColumn],
) -> InsertResult

Append one model's per-variant scores. rows holds the columns plus the variant-identity columns.

Returns an InsertResult: added is the number of rows this call newly persisted, and skipped is the number the store declined because it already held a score for that (model_id, variant_id).

A score is a finite value, or None when it is missing. A store raises ValueError for NaN or an infinity, before it persists any row of the call.

Idempotency is adapter-specific and NOT guaranteed by this contract. A store MAY dedup on re-add (the sqlite store uses INSERT OR IGNORE, so re-adding a variant lands in skipped) or MAY append unconditionally (the BigQuery store uses WRITE_APPEND, always reporting skipped=0 and physically duplicating a re-added row). Callers therefore dedup upstream — get_unscored_variants is the gate that keeps already-scored variants out of rows — and must not rely on add_scores itself to be idempotent.

primary_write

primary_write(
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    columns: list[ScoreColumn],
    lineage_descriptor: LineageDescriptor | None = None,
) -> AbstractAsyncContextManager[PrimaryScoreWriter]

Open one staged write unit for a run's primary rows: async with store.primary_write(...) as writer.

The unit yields a PrimaryScoreWriter whose add(rows) takes one batch at a time. A clean exit commits the unit; an exception aborts it and propagates. The scoring engine writes every primary row of one ingest through one unit, after the run's details and artifacts, so the primary rows stay the per-variant commit marker.

The default is a pass-through: each add is one immediate add_scores call, so a batch is durable as soon as it returns and an abort leaves the batches already added in place. It is not atomic, and atomic_primary_write stays False. A store that can stage the batches of a whole unit and publish them together (one database transaction; a staging table promoted on commit and dropped on abort) overrides this method and sets atomic_primary_write = True; where the staged rows live is the store's concern, never the caller's. A store that subclasses another and changes how add_scores writes must override this method too, or inherit a unit that bypasses its add_scores.

lineage_descriptor, the run's LineageDescriptor, is verified when the unit opens; a compact-lineage store records it when the unit registers the lineage. Other stores keep no descriptors.

read_run_identity abstractmethod async

read_run_identity(
    model_id: str,
) -> PluginRunIdentity | None

Return recorded provenance compatible with model_id scores, or None when none exist.

A store may retain more than one exact run identity when compatible wrapper releases incrementally populated the model cache; this method returns one representative immutable identity. The engine compares its role-specific scientific compatibility before treating existing rows as cache hits.

get_unscored_variants abstractmethod async

get_unscored_variants(
    variant_ids: Tabular, model_id: str
) -> Tabular

Return the variant_ids this store has no score for under model_id, so already-scored variants are not scored again. Mirrors AnnotationSource.get_unannotated_variants, which names the same input the same way.

read_score async

read_score(
    model_id: str, variant_id: str
) -> VariantRecord | None

Read one model's stored score columns for one variant. Return None if this store holds no score for that (model_id, variant_id).

This is the only read a persistence-only store has to implement. The materialize driver calls it, through read_scores, to build each variant's list of ModelScore. The return value maps the plugin's score_columns keys to their values, with boolean columns decoded back from whatever the store used to save them. The method is async so a database-backed store can await its round trip instead of blocking the event loop. The default implementation raises, so a store that overrides materialize with a single query (such as BigQuery) does not have to provide it.

read_scores async

read_scores(
    model_id: str, variant_ids: Sequence[str]
) -> dict[str, VariantRecord]

Read every stored score for model_id among variant_ids, keyed by variant_id. Variants this store has no score for are left out of the returned mapping.

The materialize driver calls this rather than read_score, so it does one read per model instead of one read per (model, variant) pair. With N variants and M models, that is M reads rather than M×N single-row round trips on the event loop. The default implementation here just calls read_score for each id. A store backed by a query engine should override it with a single variant_id IN (...) fetch, as SqliteScoreStore does. A persistence-only store can implement only read_score and inherit this.

materialize async

materialize(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
) -> AsyncIterator[MaterializedVariant]

Produce one per-variant summary for each distinct id in variant_ids, in ascending variant_id order. This is the default driver that persistence-only stores inherit.

Every store yields the same rows in the same order: exactly one MaterializedVariant per distinct requested variant — a repeated id is materialized once — ordered by canonical variant_id (plain string order, as SQL ORDER BY variant_id sorts it). Request order is not kept: a warehouse store streams its result sorted by variant_id so it can group each variant's rows without buffering the job, and a relation-scoped request has no request order to keep. A non-canonical id raises VariantIdentityError.

Only the sources that serve the models' assembly are joined (see applicable_annotation_sources): a source that declares another assembly than the one the models were scored on never annotates these variants, and a dependency only such a source supplies reads as missing.

It walks the sorted variants in chunks of _MATERIALIZE_CHUNK_VARIANTS (10,000). For each chunk it loads and merges the chunk's annotations with fetch_annotations, collects each model's scores for the chunk through read_selected_scores, and passes each variant's to materialize_variant, which does the join and prioritization the same way for every store. It yields one MaterializedVariant per variant. A persistence-only store inherits this method unchanged and implements only read_score.

The method is an async generator: a job with millions of variants must not hold the whole result in memory, so callers async for over the yielded variants, and only one chunk's annotations and scores are held at a time. A source or store error in a later chunk is therefore raised after the earlier chunks' rows were yielded. Each yielded MaterializedVariant carries model_scores, the prioritized / prioritized_model_ids / prioritized_source_names rollup, and the variant's annotations. Every store fills annotations the same way: the declared columns that have a value, with their logical types (see MaterializedVariant). The most-active-cell-type pick and the headline sort value are computed by a separate plugin-declared display rollup, not here.

A warehouse store that can do the join and prioritization in one query overrides this method; see BigQueryScoreStore. Anything specific to such a backend — for example passing a temporary-table reference when the id list is too long for one request — lives on that override, not on this driver.

export_download async

export_download(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    destination_uri: str,
    storage: Storage | None = None,
    prioritized_only: bool = False,
) -> str

Write every requested variant as one flat, wide, tab-delimited download under destination_uri (a prefix) and return that URI.

This produces a different artifact from materialize. It covers every requested variant, not only the prioritized ones, and lays out each variant as one row with the per-model score columns side by side. Its consumer is a file download, not the database rows. The columns are the wide layout of altar.plugins.data_lake.export (wide_layout), the same on every store. prioritized_only keeps only the prioritized variants, the prioritized download.

The default implementation materializes and writes the file with write_wide_csv: one gzipped part, <destination_uri>/part-000000000000.csv.gz, written through storage (the local filesystem when None). A backend that can export in bulk overrides it: BigQueryScoreStore runs one EXPORT DATA job into a gs:// prefix, with the same columns, and does not use storage. Anything specific to that backend, such as scoping the export to a temporary-table reference, lives on the override rather than in this base signature; see materialize.

export_prioritized async

export_prioritized(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    destination_uri: str,
    storage: Storage | None = None,
) -> str

Write only the prioritized variants among variant_ids as NDJSON under destination_uri (a prefix) and return that URI.

This is the database branch of the two exports: each line is one MaterializedVariant as JSON, with model_scores nested (see prioritized_record), so a loader copies a part into a database and decodes each row with one json.loads. That database load is the caller's step, not this method.

The default implementation materializes and writes <destination_uri>/part-000000000000.json.gz through storage (the local filesystem when None) with write_prioritized_ndjson. BigQueryScoreStore overrides it with one EXPORT DATA job that aggregates the same records in SQL.

result_buckets

result_buckets(
    *,
    variant_ids: Sequence[str],
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    rider: DisplayRider | None = None,
    bucket_rows: int | None = None,
    bucket_bytes: int | None = None,
) -> AbstractAsyncContextManager[ResultBuckets]

Produce the export of variant_ids as result buckets: async with store.result_buckets(...) as buckets.

The buckets (altar.plugins.data_lake.result_buckets) hold one row per distinct requested variant, in genomic order, with its annotations once and its models' scores nested in model_scores; the altar.plugins.data_lake.bucket_export writers render them to every export layout into any Storage. The scope and values are materialize's. rider, a DisplayRider, adds its columns, computed per variant. bucket_rows sets the variants per bucket, or bucket_bytes a target size (default 128 MiB) that estimated_variant_bytes turns into rows the same way on every store. The context manager releases what the store created for the export when the block exits.

The default reads each bucket with materialize over the bucket's variants, so a persistence-only store exports without implementing anything, and evaluates the rider with SQLite (evaluate_rider). A warehouse store overrides it with its own query (BigQueryScoreStore).

SqliteScoreStore

SqliteScoreStore(
    path: str = ":memory:", *, lineages: bool = False
)

Bases: ScoreStore

Score store backed by SQLite. It uses one table, variant_scores, keyed by (model_id, variant_id), with one column per plugin score column. New columns are added by ensure_schema without dropping any.

primary_write

primary_write(
    *,
    model_id: str,
    model_name: str,
    plugin_identity: PluginRunIdentity,
    columns: list[ScoreColumn],
    lineage_descriptor: LineageDescriptor | None = None,
) -> AbstractAsyncContextManager[PrimaryScoreWriter]

Write every batch of the unit in one SQLite transaction: a commit publishes all of them, an abort none.

Schema changes commit on their own, so the unit applies them before it begins the transaction. The model's schema and run identity (or, with compact lineages, the lineage declaration and its provenance) are registered inside it, by the first batch, as add_scores registers them, so an aborted unit leaves no trace of its run. While the unit is open, reads on this store see its uncommitted rows, and every other write through this store raises rather than committing the unit early. A lineage_descriptor is verified when the unit opens and, with compact lineages, recorded with the lineage inside the same transaction.

get_unscored_variants async

get_unscored_variants(
    variant_ids: Tabular, model_id: str
) -> list[str]

Return the variant_ids this store holds no score for under model_id, in input order.

It looks the IDs up in variant_id IN (...) queries of a few hundred IDs each, as read_scores does, so a call reads only the requested keys however many scores the model already has.

read_score async

read_score(
    model_id: str, variant_id: str
) -> VariantRecord | None

Return one model's stored score for a variant, or None if there is none. Calls the bulk read_scores so that the row-decoding logic lives in one place.

read_scores async

read_scores(
    model_id: str, variant_ids: Sequence[str]
) -> dict[str, VariantRecord]

Return the stored scores for model_id across the given variant_ids, keyed by variant id.

This fetches them in variant_id IN (...) queries of a few hundred ids each, so the inherited materialize driver does one bulk read per model instead of one read per (model, variant) pair, and a long id list stays under SQLite's host-parameter limit. Bool columns are converted back from their 0/1 storage. The model-specific schema is persisted alongside the score rows, so ensuring another plugin's schema cannot change the shape of this read.

close

close() -> None

Close the store's SQLite connection; a second call does nothing.

Close a file-backed store when you are done with it, or open it in a with block, which closes it on exit.

SqliteDetailStore

SqliteDetailStore(path: str = ':memory:')

Bases: DetailStore

Lossless named detail storage using schema-independent SQLite tables.

Rows are kept as canonical JSON while their parent variant, lineage, and logical-key digest are indexed. This reference layout can accept any binding schema without adding physical columns or coupling a backend to a model. One metadata row describes each lineage's table, and a provenance sidecar retains every compatible exact run identity used to populate it. A production adapter may use native typed columns while preserving the same contract.

close

close() -> None

Close the detail store's SQLite connection; a second call does nothing.

StorageArtifactStore

StorageArtifactStore(
    storage: Storage,
    root_uri: str,
    index_path: str = ":memory:",
)

Bases: ArtifactStore

Artifact files in a Storage tree under root_uri; their records in a SQLite index at index_path.

One metadata row describes each lineage's artifact, a provenance sidecar retains every compatible exact run identity that wrote under it, and a variant table indexes which variants each file covers so coverage queries never read the files. index_path may be ":memory:" for tests.

locate async

locate(record: ArtifactRecord) -> str

Return the record's URI; the bytes are readable through this store's storage.

close

close() -> None

Close the artifact index's SQLite connection; a second call does nothing.

InsertResult dataclass

InsertResult(added: int, skipped: int = 0)

The result of a score or annotation write: how many rows were added and how many were skipped because the store already held them.

Both data-lake roles return this — ScoreStore.add_scores and AnnotationSource.add_annotations — so it lives in this shared module rather than in either role's own module. Whether a write dedups at all is adapter-specific: an append-only store (for example the BigQuery score store) always reports skipped=0, while a store that ignores re-adds (the sqlite stores) counts them under skipped. See ScoreStore.add_scores for the idempotency contract.

ScoreColumn dataclass

ScoreColumn(
    name: str,
    dtype: ScalarDtype,
    label: str,
    nullable: bool = True,
)

One per-variant score column a model plugin produces.

Declaring the column here sets four things that must otherwise be kept in sync by hand: the score-store schema (table columns / DDL), the write-time validator, the wide-TSV column names, and the frontend column definitions.

It lives in this shared leaf module — not in model/plugin.py — because it is the schema the score store consumes (ScoreStore.ensure_schema / add_scores); keeping it here lets the data-lake axis name it without importing back into the model axis. model.plugin re-exports it for architecture authors.

get_artifact_store_registry

get_artifact_store_registry() -> PluginRegistry[
    type[ArtifactStore]
]

Return the shared registry of artifact-store adapters.

require_artifact_coverage

require_artifact_coverage(
    schema: ArtifactSchema,
    key: ArtifactKey,
    variant_ids: Sequence[str],
) -> None

Reject a file whose key and coverage do not fit the artifact's declared scope.

A batch file's key names no variant, since the file covers its task's whole batch. A variant file's key names the one variant the file covers, and variant_ids is exactly that variant. Every :class:ArtifactStore applies this rule in :meth:~ArtifactStore.put, so a router that collected a per-variant file as if it covered the batch, or the reverse, fails before anything is recorded.

render_long_export async

render_long_export(
    buckets: ResultBuckets,
    destination_uri: str,
    *,
    storage: Storage,
    export_format: LongExportFormat = "parquet",
    prioritized_only: bool = False,
    model_info: Mapping[str, LongModelInfo] | None = None,
    provenance: ProvenanceCallback | None = None,
) -> dict[str, str]

Write the buckets as the long layout under destination_uri, and return each table's directory.

The tables, files and values are those of write_long_export: models, variants and one scores/<table> per result schema, each one part, Parquet or (export_format="tsv") gzipped TSV. The variants table also carries the rider's columns, after prioritized, when the export has a rider. Rows are in genomic order, and a score table's rows for one variant follow the export's model order. Each table is written to local scratch as the buckets arrive and uploaded at the end.

render_occurrence_buckets async

render_occurrence_buckets(
    buckets: ResultBuckets,
    occurrence_parts: Sequence[str],
    destination_uri: str,
    *,
    storage: Storage,
    bucket_size: int,
) -> list[str]

Join the buckets' wide rows onto the ingest occurrence table, one Parquet part per record-ordinal bucket.

This is write_occurrence_buckets fed from result buckets, with the same altar-augmentation-buckets-v1 directory and the same columns (bucket_schema), rider columns included. The wide rows are spilled to a SQLite file keyed by variant as the buckets arrive, then each occurrence's row is read back in record order.

render_prioritized_ndjson async

render_prioritized_ndjson(
    buckets: ResultBuckets,
    destination_uri: str,
    *,
    storage: Storage,
    int64_as_string: bool = True,
) -> str

Write the prioritized variants as gzipped NDJSON, <destination_uri>/part-000000000000.json.gz.

Each line is the record BigQueryScoreStore.export_prioritized exports, field for field: variant_id, the rider's columns, model_scores (each entry model_id, model_name, every score field of the export, prioritized, and score_lineage_id on a compact-lineage export), prioritized_model_ids, prioritized_source_names, each annotation column in its native shape, then prioritized. An entry's model_name is the name the store holds with the score (the scores table's, on BigQuery). Records are in genomic order, and each record's model_scores in the order of the export's models (BigQuery's own export aggregates them in no set order).

Altar's NDJSON format: a missing repeated value is [], a NaN or infinite float is the string "NaN", "Infinity" or "-Infinity" (JSON has no such numbers, and a database's JSON type refuses the bare tokens), with int64_as_string (the default) an integer is a JSON string, a null field is written as null, and a float is its shortest round-trip text. BigQuery's JSON export holds the same values in other text: it omits null fields and writes floats to 15 or 17 significant digits. altar.testing.compare_prioritized_ndjson compares the two by value.

render_wide_parquet async

render_wide_parquet(
    buckets: ResultBuckets,
    destination_uri: str,
    *,
    storage: Storage,
    prioritized_only: bool = False,
    provenance: ProvenanceCallback | None = None,
) -> str

Write the buckets as the wide table in Parquet, <destination_uri>/part-000000000000.parquet.

The schema is arrow_schema(buckets.layout), rider columns included. Each bucket is one row group, so the bucket size sets the row-group size. With provenance, its mapping is stored as JSON text in the file's key-value metadata under altar.provenance.

render_wide_tsv async

render_wide_tsv(
    buckets: ResultBuckets,
    destination_uri: str,
    *,
    storage: Storage,
    prioritized_only: bool = False,
    part: str = DOWNLOAD_PART,
    provenance: ProvenanceCallback | None = None,
) -> str

Write the buckets as the gzipped, tab-delimited wide table <destination_uri>/<part>, and return the prefix.

The columns are buckets.layout.columns, with a header row, and the cells are those of write_wide_csv: NULL is empty, a boolean true/false, a repeated, struct or json value its compact JSON text. Rows are in genomic order. prioritized_only keeps only the prioritized variants. part defaults to the name a warehouse export's first shard has; altar run passes TSV_PART. With provenance, its mapping is written to <destination_uri>/_provenance.json after the last row.

get_detail_store_registry

get_detail_store_registry() -> PluginRegistry[
    type[DetailStore]
]

Return the shared registry of detail-store adapters.

column_manifest_tsv

column_manifest_tsv(
    entries: Iterable[ColumnManifestEntry],
) -> str

Render manifest entries as the text of _columns.tsv (provisional).

A header row of COLUMN_MANIFEST_FIELDS, then one tab-separated row per entry, quoted as the wide TSV quotes its cells. An unset field is empty, and repeated is true or false.

export_col_alias

export_col_alias(
    model_id: str, model_name: str, col: str
) -> str

Build the wide-export column name of one model field: the model's prefix plus the field's name.

The score-column suffix is preserved for consumers that group columns by score type.

export_column_prefix

export_column_prefix(model_id: str, model_name: str) -> str

Build a model's stable, collision-resistant column prefix for the wide exports.

The readable token comes from model_name; a digest of the stable model_id distinguishes duplicate names and names that sanitize to the same token. model_id is already required to be unique within a materialization. The augmented-VCF writer reads this prefix from its models JSON as column_prefix.

wide_column_manifest

wide_column_manifest(
    layout: WideExportLayout,
) -> list[ColumnManifestEntry]

Describe every column of layout, one entry per column in layout.columns order (provisional).

It reads only the layout, so it describes any file with these columns: the portable writers' TSV and Parquet files, an altar run export, and BigQueryScoreStore.export_download's TSV for the layout its download_header names, whose display rider columns are entries of kind display.

wide_layout

wide_layout(
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    *,
    rider_columns: Sequence[str] = (),
    score_lineages: bool = False,
    rider_dtypes: Sequence[ScalarDtype] = (),
) -> WideExportLayout

Build the wide-export layout for models and annotation_sources (see the module docstring).

rider_columns are a display rider's output columns, placed after prioritized, and rider_dtypes, when given, their scalar types in the same order. score_lineages adds each model's score_lineage_id column, for a store in compact-lineage mode. Raises SchemaCollisionError when two sources declare one column, a score column collides with an annotation column, or any two export columns share a name.

write_column_manifest async

write_column_manifest(
    layout: WideExportLayout,
    destination_uri: str,
    *,
    storage: Storage,
) -> str

Write the column manifest of layout as <destination_uri>/_columns.tsv and return its URI (provisional).

Its leading underscore keeps a part-* glob, and a dataset reader that opens the whole directory, from taking it for a data file.

write_occurrence_buckets async

write_occurrence_buckets(
    results: AsyncIterable[MaterializedVariant],
    layout: WideExportLayout,
    occurrence_parts: Sequence[str],
    destination_uri: str,
    *,
    storage: Storage,
    bucket_size: int,
) -> list[str]

LEFT JOIN results onto the ingest occurrence table and write one Parquet part per ordinal bucket.

This is the portable counterpart of BigQueryScoreStore.export_occurrence_buckets, producing the same altar-augmentation-buckets-v1 directory altar.variants.export.augmented_vcf.write_augmented_vcf reads: <destination_uri>/bucket=<n>/part-000000000000.parquet holds every occurrence whose record_ordinal divides to bucket n by bucket_size, in (record_ordinal, alt_index) order, as the occurrence columns plus result_variant_id and the wide layout's columns (bucket_schema). An occurrence whose canonical_variant_id has a result row gets that row; one with no result, or no canonical id at all, has NULL in every result column, which the augmented-VCF writer reports as NO_RESULT. Only buckets with an occurrence are written; their URIs are returned in bucket order.

occurrence_parts are the ingest's occurrence shards (<ingest>/occurrences/part-*.parquet, with OCCURRENCE_ARROW_SCHEMA), read through storage in the order given, which must be record order: a shard's ordinals follow the previous shard's, as stream_validate_variants writes them.

Neither the results nor a whole bucket is held. The result rows are first spilled to a SQLite file keyed by variant id, and each occurrence's row is read back from it. A bucket's joined rows go to a local Parquet file in row groups of _PARQUET_BATCH_ROWS as the occurrences arrive, and the file is uploaded once the ordinals move past the bucket. Both live in one temporary directory (under TMPDIR), removed when the export returns or raises. The warehouse export joins in SQL instead.

write_prioritized_ndjson async

write_prioritized_ndjson(
    results: AsyncIterable[MaterializedVariant],
    layout: WideExportLayout,
    destination_uri: str,
    *,
    storage: Storage,
) -> str

Write the prioritized variants among results as gzipped NDJSON under destination_uri and return it.

The one part is <destination_uri>/part-000000000000.json.gz, one prioritized_record per line, so a loader decodes each row with one json.loads. Rows keep the materialize order.

write_wide_csv async

write_wide_csv(
    results: AsyncIterable[MaterializedVariant],
    layout: WideExportLayout,
    destination_uri: str,
    *,
    storage: Storage,
    prioritized_only: bool = False,
    part: str = DOWNLOAD_PART,
    provenance: ProvenanceCallback | None = None,
) -> str

Write results as the gzipped, tab-delimited wide download under destination_uri and return that URI.

The one part is <destination_uri>/<part>, part-000000000000.csv.gz by default, the name a warehouse export's first shard has, with a header row of layout.columns. altar run passes TSV_PART instead, so its file is named for what it holds. prioritized_only keeps only the prioritized variants, the prioritized download. Row order is the materialize order, ascending variant_id.

With provenance, the mapping it returns after the last row is written as indented JSON to <destination_uri>/_provenance.json.

write_wide_parquet async

write_wide_parquet(
    results: AsyncIterable[MaterializedVariant],
    layout: WideExportLayout,
    destination_uri: str,
    *,
    storage: Storage,
    prioritized_only: bool = False,
    provenance: ProvenanceCallback | None = None,
) -> str

Write results as the wide table in Parquet under destination_uri and return that URI.

The one part is <destination_uri>/part-000000000000.parquet with arrow_schema(layout): repeated and struct annotations keep their native types, so nothing is flattened to text except json columns. prioritized_only keeps only the prioritized variants. Rows are written a row group at a time as results yields them, in the same row groups write_parquet_part writes, so the whole result is never held.

With provenance, the mapping it returns after the last row is stored as compact JSON text in the file's key-value metadata under PROVENANCE_METADATA_KEY (altar.provenance). The columns are unchanged.

long_score_tables

long_score_tables(
    layout: WideExportLayout,
) -> tuple[LongScoreTable, ...]

Group the layout's models into score tables by plugin ID and result schema version (provisional).

Tables come in the order of their first model, and each lists its models in layout order. Raises SchemaCollisionError when a score field is named variant_id, model_id, score_lineage_id or prioritized, the key columns of every score table.

write_long_export async

write_long_export(
    results: AsyncIterable[MaterializedVariant],
    layout: WideExportLayout,
    destination_uri: str,
    *,
    storage: Storage,
    export_format: LongExportFormat = "parquet",
    prioritized_only: bool = False,
    model_info: Mapping[str, LongModelInfo] | None = None,
    provenance: ProvenanceCallback | None = None,
) -> dict[str, str]

Write results as the long export under destination_uri and return each table's URI (provisional).

The tables are models, variants and one scores/<table> per result schema (see the module docstring and long_score_tables), each a directory holding one part, part-000000000000.parquet or, with export_format="tsv", part-000000000000.tsv.gz with a header row. Parquet keeps the declared types; TSV writes repeated, struct and json values as JSON text, as the wide TSV does. prioritized_only keeps only the prioritized variants, and their score rows. model_info adds, by model key, what the layout does not know to the models table. With provenance, the mapping it returns after the last row is written as indented JSON to <destination_uri>/_provenance.json.

The returned mapping is keyed by each table's path relative to destination_uri: models, variants, then scores/<table> in long_score_tables order. Rows keep the materialize order, ascending variant_id, and a score table's rows for one variant follow the layout's model order. A layout with display-rider columns is refused, since a rider is computed only by a warehouse query.

annotation_prioritizers_from

annotation_prioritizers_from(
    sources: Iterable[Any],
) -> list[AnnotationPrioritizer]

Build an AnnotationPrioritizer for each source that can prioritize a variant.

A source can prioritize when its prioritize_predicate() returns a predicate rather than None; sources that return None only annotate and are skipped. Each source is used by duck typing — its .name and .prioritize_predicate() — so AnnotationSource does not have to be imported here.

materialize_variant

materialize_variant(
    variant_id: str,
    model_scores: Sequence[ModelScore],
    annotation: VariantRecord,
    annotation_prioritizers: Sequence[
        AnnotationPrioritizer
    ] = (),
) -> MaterializedVariant

Combine one variant's per-model scores and annotations into a MaterializedVariant.

The steps:

  • Per-model prioritization: evaluate each model plugin's predicate over the variant's scores merged with its annotations. Keep the three-valued result (True, False, or None) so that a missing input yields None rather than a forced False.
  • model_scores: one to_model_score entry per model, with that model's three-valued prioritized included. to_model_score turns a None into False.
  • prioritized_model_ids: the ids of the models whose entry is prioritized.
  • Model-less prioritizers: evaluate each AnnotationPrioritizer's predicate over the variant's annotations. A missing column makes the predicate None, so it does not fire. The ones that do fire are collected in prioritized_source_names.
  • prioritized: true when any model or any prioritizing source is True, otherwise False.
  • annotations: the columns of annotation that have a value (see present_annotations).

The most-active cell type and the headline sort magnitude are not computed here.

bucket_export_layout

bucket_export_layout(
    models: Sequence[ScoredModel],
    annotation_sources: Sequence[AnnotationSource],
    *,
    rider: DisplayRider | None = None,
    score_lineages: bool = False,
) -> WideExportLayout

Build the wide layout of a result-bucket export: wide_layout with the rider's typed columns (provisional).

Pass only the sources that apply to the models (applicable_annotation_sources). The rider's column types come from rider_column_dtypes.

bucket_rows_for

bucket_rows_for(
    layout: WideExportLayout,
    *,
    bucket_rows: int | None = None,
    bucket_bytes: int | None = None,
) -> int

Return the variants per bucket: bucket_rows, or bucket_bytes (default 128 MiB) over the estimate.

genomic_sort_key

genomic_sort_key(
    variant_id: str,
) -> tuple[int, bytes, int, bytes, bytes]

Return the key that orders canonical variant ids in genomic order (provisional).

The key is the chromosome's rank (chromosome_rank), its name, the position, then the reference and alternate alleles, names and alleles compared as bytes. Distinct canonical ids get distinct keys. Raises VariantIdentityError for a non-canonical id.

plan_buckets

plan_buckets(
    row_count: int, bucket_rows: int
) -> tuple[ResultBucketDescriptor, ...]

Cut row_count ordinals into consecutive buckets of bucket_rows, the last one possibly shorter.

result_bucket_schema

result_bucket_schema(layout: WideExportLayout) -> Schema

Return the Arrow schema of a result bucket for layout (provisional; see the module docstring).

get_score_store_registry

get_score_store_registry() -> PluginRegistry[
    type[ScoreStore]
]

Return the registry of score-store plugins registered under altar.score_stores.

safe_identifier

safe_identifier(name: str) -> str

Check that name is a valid identifier before it is placed into a SQL statement, and return it.

Score column names come from plugin.score_columns() in code, but a store that builds SQL by hand still validates the name rather than trusting it. Raises ValueError if the name is not a plain identifier. Used by the stores that build SQL as strings (sqlite and BigQuery).