Skip to content

Evidence sources

altar.sources defines annotation sources and variant–gene link sources. Evidence bindings and applications that serve their own tables import from it. Add an annotation source shows how to write one.

Stable, except the published-dataset, dataset-builder, link-dataset and position-score names, which are provisional. Provisional names lists them.

sources

Stable annotation-source and variant-to-gene evidence contracts.

This is the supported import surface for bindings backed by precomputed data, warehouses, or services. The exported records retain source evidence without forcing it into a model result.

ANNOTATION_IDENTITY_COLUMN module-attribute

ANNOTATION_IDENTITY_COLUMN = '_altar_annotation_identity'

The column a writable lake stamps with its source's AnnotationSourceIdentity.identity_hash.

An identity-scoped BigQueryAnnotationSource writes it on every row and deduplicates and reads only rows that carry its own hash, so a data release bump re-annotates instead of serving cached values of the old release. BigQueryAnnotationSource.add_annotations_from_uri stamps it on files it loads; a host that loads such a lake with its own job stamps the same column. No annotation column may use this name.

LINK_DATASET_METADATA = ParquetReleaseMetadata(
    prefix="altar_links.",
    distribution="altar",
    label="variant-gene link",
    command="a link dataset builder (write_link_dataset)",
    layout="variant-gene-links/1",
)

How every file of a link dataset records its release: the fields link_dataset_release_fields names, under the altar_links. prefix, and the layout name and version, variant-gene-links/1. Provisional.

BUILDER_VERSION_METADATA_FIELD module-attribute

BUILDER_VERSION_METADATA_FIELD = 'builder_version'

The metadata field that records the installed version of the binding that built a dataset.

DEFAULT_INPUT_ATTEMPTS module-attribute

DEFAULT_INPUT_ATTEMPTS = 3

How many times a builder reads a remote input, or uploads a built file, before it gives up (BuildOutput).

DEFAULT_STDIN_NAME module-attribute

DEFAULT_STDIN_NAME = 'stdin'

The output name standard input takes when the caller gives none.

GENOME_BUILD_METADATA_FIELD module-attribute

GENOME_BUILD_METADATA_FIELD = 'genome_build'

The metadata field that records the genome build of a dataset's coordinates.

LAYOUT_METADATA_FIELD module-attribute

LAYOUT_METADATA_FIELD = 'layout'

The metadata field that records which of a binding's file layouts a dataset has (ParquetReleaseMetadata.layout).

RELEASE_METADATA_FIELD module-attribute

RELEASE_METADATA_FIELD = 'release'

The metadata field that records the upstream release a dataset was built from.

STDIN module-attribute

STDIN = '-'

The input name that reads standard input, so a host can stream a VCF from storage the builder never opens.

VARIANT_ID_COLUMN module-attribute

VARIANT_ID_COLUMN = 'variant_id'

The column every built file keys its rows by; by default check_unique_keys rejects a value that occurs twice.

POSITION_SCORE_METADATA module-attribute

POSITION_SCORE_METADATA = ParquetReleaseMetadata(
    prefix="altar_positions.",
    distribution="altar",
    label="position score",
    command="the track's position score builder",
    layout="position-scores/1",
)

How every file of a position score dataset records its release: the fields position_score_release_fields names, under the altar_positions. prefix, and the layout name and version, position-scores/1. Provisional.

LINK_GENE_FIELDS = (
    AnnotationColumn(
        name="gene_id", dtype="str", label="Gene ID"
    ),
    AnnotationColumn(
        name="gene_namespace",
        dtype="str",
        label="Gene Namespace",
    ),
    AnnotationColumn(
        name="gene_symbol", dtype="str", label="Gene Symbol"
    ),
    AnnotationColumn(
        name="score", dtype="float", label="Link Score"
    ),
    AnnotationColumn(
        name="score_direction",
        dtype="str",
        label="Score Direction",
    ),
    AnnotationColumn(
        name="biosample", dtype="str", label="Biosample"
    ),
    AnnotationColumn(
        name="biosample_id",
        dtype="str",
        label="Biosample ID",
    ),
    AnnotationColumn(
        name="regulatory_element_id",
        dtype="str",
        label="Regulatory Element ID",
    ),
    AnnotationColumn(
        name="element_type",
        dtype="str",
        label="Element Type",
    ),
    AnnotationColumn(
        name="source_id", dtype="str", label="Source ID"
    ),
    AnnotationColumn(
        name="source_name", dtype="str", label="Source Name"
    ),
    AnnotationColumn(
        name="method", dtype="str", label="Method"
    ),
    AnnotationColumn(
        name="data_release",
        dtype="str",
        label="Data Release",
    ),
    AnnotationColumn(
        name="record_id",
        dtype="str",
        label="Link Record ID",
    ),
    AnnotationColumn(
        name="source_record_id",
        dtype="str",
        label="Source Record ID",
    ),
    AnnotationColumn(
        name="source_dataset",
        dtype="str",
        label="Source Dataset",
    ),
)

The fields of one link_genes record: the link's gene, score, biosample and provenance.

VARIANT_GENE_LINK_ANNOTATION_CONTRACT = AnnotationContract(
    source_id=VARIANT_GENE_LINK_ANNOTATION_SOURCE_ID,
    schema_version="1.0.0",
    columns=(
        AnnotationColumn(
            name="link_top_gene",
            dtype="str",
            label="Top Linked Gene",
        ),
        AnnotationColumn(
            name="link_top_gene_symbol",
            dtype="str",
            label="Top Linked Gene Symbol",
        ),
        AnnotationColumn(
            name="link_top_score",
            dtype="float",
            label="Top Link Score",
        ),
        AnnotationColumn(
            name="link_gene_count",
            dtype="int",
            label="Linked Gene Count",
        ),
        AnnotationColumn(
            name="link_count",
            dtype="int",
            label="Link Count",
        ),
        AnnotationColumn(
            name="link_genes",
            dtype="struct",
            label="Linked Genes",
            repeated=True,
            fields=LINK_GENE_FIELDS,
        ),
    ),
)

The variant-gene link annotation columns, version 1.0.0.

link_genes holds one record per link, strongest first (see project_variant_gene_links), so its first element is the top link. link_top_gene is that link's gene ID, link_top_gene_symbol its symbol when the source recorded one, and link_top_score its score; the three are absent when no link has a usable score. link_gene_count counts the distinct linked gene IDs and link_count the link records. A variant with no link has no row.

VariantIdentityError

Bases: ValueError

A variant key is malformed, non-canonical, or disagrees with its locus.

VariantKey dataclass

VariantKey(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
)

Canonical one-based biallelic key serialized as chr:pos:ref:alt.

Fields follow the rules in this module's documentation. The class normalizes spelling but does not normalize biological representation. Callers must left-align and trim indels against the selected reference before constructing a key when equivalence across representations matters.

variant_id property

variant_id: str

Return the portable score, detail, and annotation join key.

parse classmethod

parse(variant_id: str) -> VariantKey

Parse and normalize a four-field textual key.

require_canonical classmethod

require_canonical(variant_id: str) -> VariantKey

Parse variant_id and reject aliases instead of silently rewriting them.

from_fields classmethod

from_fields(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
) -> VariantKey

Construct a canonical key from decomposed one-based locus fields.

AllelicKey dataclass

AllelicKey(
    variant_id: str,
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
)

One backend-ready biallelic variant key.

variant_id is the caller's stable identity and is returned unchanged. The remaining fields are the values used by the backend's physical lookup; a binding may therefore add or remove a chromosome prefix while retaining the original variant_id. allelic_key fills them with the minimal representation of variant_id, so two keys with different variant_id values can share one physical key.

AllelicKeyColumns dataclass

AllelicKeyColumns(
    chromosome: str,
    position: str,
    reference_allele: str,
    alternate_allele: str,
)

Physical column names used to join a record collection to requested variants.

as_tuple

as_tuple() -> tuple[str, str, str, str]

Return key columns in chromosome, position, reference, alternate order.

AllelicRecord dataclass

AllelicRecord(
    variant_id: str, fields: Mapping[str, object]
)

One source-native record matched to the caller's variant_id.

AllelicRecordBackend

Bases: Protocol

Retrieve source-native records without owning their biological interpretation.

fetch async

fetch(query: AllelicRecordQuery) -> Sequence[AllelicRecord]

Return all matching records, restricted to query.keys and query.fields.

Zero, one, or many records may match each variant. Returned variant_id values must come from the request, fields must equal the requested projection, and values must not be aggregated or reinterpreted. Two keys with different variant_id values and one physical key each get every matching record.

AllelicRecordQuery dataclass

AllelicRecordQuery(
    keys: tuple[AllelicKey, ...], fields: tuple[str, ...]
)

A projected, batch lookup against one source-native record collection.

ParquetAllelicRecordBackend

ParquetAllelicRecordBackend(
    dataset: str | PathLike[str],
    *,
    key_columns: AllelicKeyColumns,
    hive_partitioning: bool = True,
    storage_options: Mapping[str, Any] | None = None,
    batch_size: int = DEFAULT_PARQUET_FETCH_BATCH_SIZE,
    max_concurrent_files: int = DEFAULT_PARQUET_MAX_CONCURRENT_FILES,
)

Query any Parquet record collection through DuckDB, on the local disk or where it is hosted.

The adapter knows only the physical allelic key columns. Projected fields are supplied per request by the scientific binding, so one instance shape works for transcript-level AlphaMissense rows, gene-level SpliceAI rows, and future precomputed sources.

dataset is one Parquet file or a directory of them, given as a local path or as a URI: file://, or a remote gs://, s3://, az://, http(s)://, hf:// or memory:// URI that an fsspec filesystem reads in place. storage_options is passed to that filesystem unchanged, for non-secret options such as {"token": "anon"} or a requester-pays project; provider credentials stay ambient. A local dataset needs no fsspec and ignores storage_options. dataset is kept as a Path for a local dataset and as the URI otherwise.

A lookup reads only what its keys can match. With hive_partitioning, a file under a <chromosome column>=<value> directory is opened only when a key is on that chromosome, and is searched for that chromosome's keys alone. Each file is scanned for its keys' chromosomes and exact positions, so Parquet row-group statistics skip every row group that holds none of them; the builders write rows sorted by position within a chromosome, which keeps those statistics tight, and in small row groups, so a point lookup reads one small row group per hit. The exact allelic key join decides which rows match.

A fetch reads its keys in consecutive batches of at most batch_size, so a cohort-scale request never becomes one query with a parameter per position. Within a batch, each file the keys can be in is read by its own query, up to max_concurrent_files at once, so a lookup spread over many files waits for the slowest file rather than for the sum of them. Records come back in request order across batches and files; a key matched in several files has its records in the files' sorted order.

One DuckDB database serves every fetch of an instance, and concurrent fetches are safe. A remote dataset is treated as an unchanging release: its file list and each file's size are fetched once per instance, and each file's footer and every byte range already read are reused by later queries rather than fetched again, so the batches after the first, and later fetches, read only row groups they have not read. A local dataset is listed on every fetch, as before.

fetch async

fetch(query: AllelicRecordQuery) -> list[AllelicRecord]

Return requested native records without blocking the event loop.

close

close() -> None

Close the DuckDB database the backend opened, if any; a second call does nothing.

A source that built this backend closes it from its own close. A later fetch opens a new database.

SqliteAllelicRecordBackend

SqliteAllelicRecordBackend(
    database: str | Path,
    *,
    table: str,
    key_columns: AllelicKeyColumns,
)

Query any SQLite table containing source-native allelic records.

A fresh connection is opened for each fetch on its worker thread, which keeps the adapter safe to call from an async application without sharing a synchronous sqlite3.Connection across threads.

fetch async

fetch(query: AllelicRecordQuery) -> list[AllelicRecord]

Return requested native records without blocking the event loop.

AnnotationDependencyError

Bases: ValueError

Raised when the configured annotation sources cannot satisfy a model's annotation dependency.

AnnotationGenomeBuildError

Bases: AnnotationDependencyError

Raised when annotation sources, or sources and models, declare different genome builds.

AnnotationUnavailableWarning

Bases: UserWarning

Emitted when sources are not joined because they serve another assembly than the variants.

It names the dropped sources, whose columns read as missing. It is also emitted for a model's dependency that no source can supply on the variants' assembly, because the source classes implementing its contract all leave that assembly out; the model's prioritization then treats the columns as unknown. Configure a source that serves the variants' assembly to supply them.

ResolvedAnnotationDependency dataclass

ResolvedAnnotationDependency(
    model_id: str,
    dependency: AnnotationDependency,
    source_names: tuple[str, ...],
    identity: AnnotationSourceIdentity | None,
    unavailable: tuple[str, ...] = (),
)

How one model's AnnotationDependency is satisfied.

source_names are the configured sources that supply the dependency's columns, in configuration order. identity is the identity of the source that declares the dependency's contract, or None when the columns were matched by name only (an UndeclaredAnnotationContractWarning was emitted) or are unavailable. unavailable names the dependency columns no source can supply on the variants' assembly; they read as missing (an AnnotationUnavailableWarning was emitted).

UndeclaredAnnotationContractWarning

Bases: UserWarning

Emitted when a dependency's columns are supplied by sources that do not declare its contract.

The columns are used by name, so materialization proceeds, but nothing states that their values follow the contract the model depends on. Declare it with identity=CONTRACT.identity(release=..., genome_build=...) on a generic source, or by overriding AnnotationSource.identity().

AnnotationColumn dataclass

AnnotationColumn(
    name: str,
    dtype: AnnotationDtype,
    label: str,
    repeated: bool = False,
    fields: tuple[AnnotationColumn, ...] = (),
)

One per-variant annotation column a source provides, such as region_type or ccre_id.

dtype, together with repeated and fields, gives the column's logical type; each backend maps that type to its own physical form. A column with repeated=True is an array of dtype. A column with dtype="struct" is a record whose fields are declared in fields, usually with repeated=True for an array of records such as nearest_genes. BigQuery stores these as native types (ARRAY<STRING>, ARRAY<STRUCT<…>>). A backend without native array or record support, such as SQLite, stores them as a JSON blob instead. That JSON form is the backend's substitute, not the column's logical type.

AnnotationContract dataclass

AnnotationContract(
    source_id: str,
    schema_version: str,
    columns: tuple[AnnotationColumn, ...],
)

A published, versioned column set that any source may implement.

source_id is a globally namespaced name, such as org.kundajelab.altar.annotation.regions, and is what a model's AnnotationDependency.source_id refers to. schema_version is a semantic version: a source whose contract has the same major version and an equal or later minor version can satisfy a dependency, because minor versions only add columns. columns is the column set every implementation declares.

A contract does not name a data release or a genome build. Several sources can implement one contract, such as a shipped binding and a host's own warehouse table; each states the release and build it serves through identity(release=..., genome_build=...).

identity

identity(
    *, release: str, genome_build: str
) -> AnnotationSourceIdentity

Return the identity of a source that serves release of this contract on genome_build.

AnnotationIdentityError

Bases: ValueError

Raised when a writable annotation lake would mix, or cannot vouch for, an annotation source identity.

Examples: a row carries another identity's hash; a SQLite lake records a different identity than the source declares; a source that is not identity-scoped writes to a scoped lake; or a lake written before identities were stored is opened by an identity-declaring source and has not yet been adopted with backfill_annotation_identity().

AnnotationSource

Bases: ABC

A per-variant reference annotation dataset.

Adapters register under altar.annotation_sources and are looked up by name. name is the registry key, such as "variant_annotations", "alphamissense", or "none".

annotate returns annotations in a portable shape, AnnotationTable (variant_id -> {column: value}), which merge_annotations and fetch_annotations consume. The BigQuery source returns a pandas frame instead and usually fuses its read into the store's query through the SqlAnnotationSource capability, skipping the portable merge. Core carries no pandas dependency, so the signatures at this boundary use Tabular, a documented alias for Any, and the conformance suite checks the actual behavior.

A source does not have to be registered as an entry point. A host can subclass this class, or configure a generic BigQueryAnnotationSource or SqliteAnnotationSource over its own tables, and pass the instance to materialization next to shipped bindings. Altar checks, joins and prioritizes on the declared annotation_columns() and identity(), never on the class or the registry name.

supported_genome_builds class-attribute

supported_genome_builds: frozenset[str] | None = None

The assemblies an instance of this class can serve, or None when the class does not restrict them.

Each instance names the one assembly it serves in identity(). Materialization reads this declaration, with annotation_contract(), to tell a dependency that no source can supply on the variants' assembly, such as region annotations for mouse variants, from one the host forgot to configure (see resolve_annotation_dependencies). The UCSC and GRC spellings of an assembly match.

option_descriptions class-attribute

option_descriptions: Mapping[str, str] = {}

What each option of from_options means, by option name, as altar sources shows it.

Provisional: the hook is expected to change.

Each key names a keyword parameter of from_options, which AnnotationSourceContract checks. A value is a short phrase in lower case, such as "the copy the builder wrote, as a path or URI". An option left out keeps the description altar sources gives every source, such as the one for genome_build, or none. A description is not part of the source's identity.

from_options classmethod

from_options(**options: Any) -> Self

Build this source from the plain values a run spec's [sources.<name>.options] mapping carries.

altar.run resolves a source by its registry name and calls this instead of the constructor, so a source is declarable from a spec only when strings, numbers, booleans and paths are enough to build it: the shipped Parquet-backed bindings take the path of a dataset their builder wrote and read the release and genome build from its metadata, and RegionAnnotationSource takes its data directory.

The default raises CapabilityNotSupported: this source's constructor needs a client, a backend, or typed column values that a spec cannot carry, so build it in code and pass it to altar.run.run as annotation_sources.

published_dataset classmethod

published_dataset() -> PublishedDataset | None

Return where this binding's publisher hosts a ready-made copy of its latest release, or None.

Provisional, like PublishedDataset: the hook is expected to change along with it.

A binding whose data is openly licensed and expensive to build may declare the copy its publisher hosts, so a user reads that release in place instead of building it, and generic tooling finds the copy through the registry without importing the binding. The copy is the binding's own builder layout, and its identity is the identity a source serving that release declares, whether it reads the published copy or an owned copy of it. A source that reads it by default does so from from_options when no owned dataset is named, resolving its options with PublishedDataset.select_copy; altar.testing.PublishedDatasetContract checks the whole arrangement.

The default None means the binding publishes nothing: the user builds or brings the data.

annotation_contract classmethod

annotation_contract() -> AnnotationContract | None

Return the AnnotationContract every instance of this class implements, or None.

Provisional: the hook is expected to change.

It lets tooling describe a registered source without building it, which may need data, credentials or the network: altar sources lists the contract's columns from it. A class that returns a contract promises that each instance's identity() names that contract and its annotation_columns() include the contract's columns, which altar.testing.AnnotationSourceContract checks. The default None means the class declares no single contract, as a generic source whose columns come from its options does.

annotation_columns abstractmethod

annotation_columns() -> list[AnnotationColumn]

Return the columns this source adds to each materialized variant row.

Mirrors ModelPlugin.score_columns() on the model axis: each producer names the columns it contributes by its role (score_columns / annotation_columns).

annotate abstractmethod async

annotate(variant_ids: Sequence[str]) -> Tabular

Fetch annotations for variant_ids and return them keyed by variant_id.

The result is a Mapping[variant_id, Mapping[column, value]], the shape merge_annotations and fetch_annotations consume. A variant this source has no row for is left out of the result; the later left-merge fills it with {}.

identity

identity() -> AnnotationSourceIdentity | None

Return the contract, data release and genome build this source serves, or None.

A source that implements a published AnnotationContract should return CONTRACT.identity(release=..., genome_build=...), and its annotation_columns() should include the contract's columns. Materialization resolves each model's AnnotationDependency against these identities (see resolve_annotation_dependencies). The default None means the source declares no contract: its columns can still satisfy a dependency by name, with an UndeclaredAnnotationContractWarning.

add_annotations async

add_annotations(rows: Tabular) -> InsertResult

Write annotation rows into this source.

The default raises CapabilityNotSupported: most sources are read-only reference datasets that core never writes to. Only a writable ingest source overrides this.

get_unannotated_variants async

get_unannotated_variants(variant_ids: Tabular) -> Tabular

Return the subset of variant_ids this source has no annotation row for.

A writable source uses this before ingest so it does not re-annotate variants it already has, the same role get_unscored_variants plays for scores. The portable contract takes variant_ids in and returns the unannotated variant_ids. A writable source that declares an identity() counts only rows written under that identity, so a new data release reports every variant unannotated (altar.testing.WritableAnnotationSourceContract checks this). A source may omit ids its write path can never store, such as a non-canonical variant_id, since reporting them would ask every job to annotate them again.

Warehouse-scale scoping — pruning against a temp-table relation or a chr clustering key instead of binding millions of ids as an array parameter — is a SQL concern, not part of this backend-neutral contract. A SQL-backed source widens this signature with those parameters on its own concrete method (BigQueryAnnotationSource.get_unannotated_variants takes variants_relation / chromosomes), the same way ScoreStore.get_unscored_variants keeps its variants_relation off the base and only the BigQuery store declares it. The temp-table refs and clustering keys stay behind the SQL adapter.

The default raises CapabilityNotSupported. A precomputed reference dataset such as AlphaMissense or CADD is never written through core, so it has nothing to deduplicate; only a writable ingest source, paired with add_annotations, overrides this.

prioritize_predicate

prioritize_predicate() -> Predicate | None

Return the predicate that gates the prioritized flag, or None.

A source that prioritizes variants, such as AlphaMissense, returns a Predicate evaluated over its own per-variant annotation columns. A source that only contributes columns without driving prioritization, such as region_type or a gnomAD frequency, returns None (the default).

This is the same Predicate type a ModelPlugin uses. It renders to SQL for the store's materialize step or to Python for the reference driver, so a source can gate prioritized with no model, model_id, or scoring job. Any aggregation from per-transcript rows down to one row per variant happens inside annotate(); the predicate reads the already-aggregated column.

close

close() -> None

Release the connections, clients and readers this source opened itself.

The default does nothing. A source that opens a connection or creates a client, for example in from_options, overrides it, and a second call must do nothing. A client or backend passed to the constructor stays open: whoever created it closes it. altar.run closes the sources it builds from a run spec when the run ends, and never one passed to it as annotation_sources.

AnnotationSourceIdentity dataclass

AnnotationSourceIdentity(
    source_id: str,
    schema_version: str,
    release: str,
    genome_build: str,
)

What a concrete source's values mean: the contract it implements plus the data release and build.

source_id and schema_version name the AnnotationContract. release is the source's data release label, such as "4.1.1-genomes"; it must change whenever the values the source returns could change. genome_build is the assembly of the variant IDs the source answers for, spelled as in score identity ("hg38"). Surrounding whitespace is stripped from release and genome_build, and both must be non-empty.

identity_hash is a deterministic hash of the four fields, suitable as a cache or deduplication key.

identity_hash property

identity_hash: str

Return sha256:<hex> over the canonical JSON of the four identity fields.

to_dict

to_dict() -> dict[str, str]

Return the four identity fields as a JSON-ready mapping.

from_dict classmethod

from_dict(
    data: Mapping[str, Any],
) -> AnnotationSourceIdentity

Rebuild an identity from to_dict() output, rejecting missing or unknown fields.

IdentityBackfill dataclass

IdentityBackfill(
    stamped: int, deleted: int, non_canonical: int = 0
)

What a writable lake's backfill changed: rows stamped, and rows deleted.

Returned by backfill_annotation_identity on BigQueryAnnotationSource and SqliteAnnotationSource, and by BigQueryAnnotationSource.backfill_identity_columns (the locus columns).

non_canonical counts the legacy rows a BigQuery backfill left unstamped because their variant_id is not canonical, such as chr9:100:C:C (REF equal to ALT) or 9:100:C:T. No canonical lookup reaches them, so they are never served or counted; find and delete them. A SQLite backfill adopts the file as a whole and always reports 0.

UnscopedAnnotationReadWarning

Bases: UserWarning

Emitted when a source that is not identity-scoped reads a table that has ANNOTATION_IDENTITY_COLUMN.

An identity-scoped writer stamps that column, so the table may hold rows of several identities, such as two data releases. A source that is not scoped reads and counts all of them: a variant can be served an arbitrary release's values, and deduplication counts a variant annotated under another release as annotated. Give the source the identity it serves and identity_scoped=True; a reader with its own relation= must also filter with the {identity_filter} token.

BigQueryAnnotationSource checks its table's schema for the column before it first reads or deduplicates against the table. It does not check a read through relation=, whose tables only the SQL names. Writes are not warned about: an unscoped write to a scoped lake raises AnnotationIdentityError.

BigQueryAnnotationSource

BigQueryAnnotationSource(
    client: Any,
    table_id: str,
    columns: Sequence[AnnotationColumn],
    *,
    name: str | None = None,
    prioritize_predicate: Predicate | None = None,
    relation: str | None = None,
    genome_default: str | None = None,
    identity: AnnotationSourceIdentity | None = None,
    identity_scoped: bool = False,
)

Bases: SqlAnnotationSource

Your own per-variant values, from a BigQuery table. Build it in code with a client and the table's columns.

It reads one table (project.dataset.table) keyed on variant_id, with one column per injected AnnotationColumn. It is a prioritizing source when a prioritize_predicate is injected; the passive/prioritizing distinction is configuration, not a subclass.

It shares the store's SQL backend, so it implements SqlAnnotationSource: the BigQuery store fuses its sql_join JoinSpec into the single materialize query instead of round-tripping through annotate().

A source constructed with identity_scoped=True (which needs an identity) is an identity-scoped lake. Every row it writes carries the identity's identity_hash in the _altar_annotation_identity column (ANNOTATION_IDENTITY_COLUMN). Deduplication counts only rows of that hash, and reads select only rows of that hash: the bare table through a filtering subquery, and a relation= through its required {identity_filter} token. Bumping the identity's release therefore re-annotates variants instead of serving the old release's cached values, and a table holding several releases still yields one row per variant. Rows written before the table carried an identity have a NULL hash and are neither served nor counted until backfill_annotation_identity() adopts them. Without identity_scoped, identity is metadata only.

Every source object that writes or reads the lake must be scoped the same way. A source that is not scoped looks up its table's schema, a metadata request that bills nothing, and treats a table with the identity column as a scoped lake. add_annotations and add_annotations_frame then raise AnnotationIdentityError instead of writing rows without a hash. The source emits UnscopedAnnotationReadWarning, once per source object, before it first reads the table (annotate and the store's fused queries) or deduplicates against it (get_unannotated_variants and export_unannotated). A read through relation= is not checked, because only its SQL names the tables it reads; deduplication and writes always use the bare table, so they are checked. If the lookup fails, the failure is logged and the operation proceeds unchecked.

client is a google.cloud.bigquery.Client (or a compatible fake). table_id is "project.dataset.table". columns and prioritize_predicate are the wiring that makes this a passive or prioritizing source. name is the logical source identity recorded in prioritized_source_names (e.g. "alphamissense"), which is distinct from the class registry key ("bigquery") and defaults to it. The client is injected so this module needs no google-cloud-bigquery at import.

relation is an optional SQL relation that overrides the bare table_id: a subquery yielding one row per variant_id and exposing at least the declared columns. It is how a source whose backing rows are per-transcript (AlphaMissense and REVEL collapse to the most-damaging transcript) or that needs computed or renamed columns (gnomAD af → af_global, TO_JSON_STRING(...) blobs) fuses in-SQL, keeping the aggregation and projection inside the source rather than the store. A {variant_filter} token in the string expands, in both sql_join and annotate, to variant_id IN UNNEST(@<param>), so the reference table is pruned to the job's variants before aggregating. A {identity_filter} token expands to _altar_annotation_identity = '<identity_hash>', so a relation over a lake holding several releases reads only this source's rows. It qualifies like {variant_filter} does (a.{identity_filter}). An identity-scoped source's relation must contain it, and any other source's relation must not. None uses the bare per-variant table.

genome_default is the genome build of a writable ingest lake: the genome that add_annotations and add_annotations_frame stamp on rows that carry none (a row naming another build is rejected), and that both get_unannotated_variants paths and export_unannotated filter on, so the dedup only counts rows of this build and cluster-prunes the annotation table's [genome, chr, variant_id] key. No build is assumed: those methods raise ValueError when it is None, rather than silently deduplicating an hg19 lake against hg38 rows. It is inert for a read-only reference source, which is never written or deduplicated.

identity declares the AnnotationContract this table implements and the data release and genome build it holds (see AnnotationSource.identity()); a host table that implements a published contract passes CONTRACT.identity(release=..., genome_build=...). On its own it is metadata: it does not change how rows are written, deduplicated or read. identity.genome_build is the build label that dependency resolution compares with score runs (for example "hg38"), while genome_default is how this table spells the build in its stored genome column (Altar's VCF ingest writes "GRCh38"). The two are independent, and neither is derived from the other.

identity_scoped=True makes the identity decide which rows this source writes, counts and reads (see the class docstring). It needs identity, and a relation must then contain {identity_filter}; both are checked here and raise ValueError. Writes stamp identity.identity_hash, and a row that carries another hash raises AnnotationIdentityError.

client property writable

client: Any

Return the BigQuery client this source runs its queries and load jobs with.

It is read-only. Assigning the client the source already holds is accepted; any other value raises AttributeError.

table_id property writable

table_id: str

Return the annotation table, as "project.dataset.table".

A source that reads through relation= still writes to and deduplicates against this table. It is read-only. Assigning the ID the source already holds is accepted; any other value raises AttributeError.

connect classmethod

connect(
    table_id: str,
    columns: Sequence[AnnotationColumn],
    *,
    name: str | None = None,
    prioritize_predicate: Predicate | None = None,
    relation: str | None = None,
    genome_default: str | None = None,
    credentials: Any = None,
    identity: AnnotationSourceIdentity | None = None,
    identity_scoped: bool = False,
) -> BigQueryAnnotationSource

Build a source against a real BigQuery client. Requires the altar[bigquery] extra.

The source owns the client it builds here, and close releases it.

close

close() -> None

Close the BigQuery client connect built; a second call does nothing.

A client passed to the constructor is left open: whoever created it closes it.

sql_join

sql_join(ctx: SqlJoinContext) -> JoinSpec

Fuse this source into the store's materialize query as a LEFT JOIN on ctx.join_key. A BigQuery source always shares the store's backend, so it always fuses and never returns None. relation is the backtick-quoted table id for a per-variant table (the store projects each injected column), or the injected aggregating or computed subquery (see __init__'s relation and JoinSpec), pruned to ctx.variants_relation (temp table) or ctx.variant_ids_param (inline param). The alias derives from the logical source name (alphamissense, cadd, …), which the store keeps distinct across sources.

ensure_schema async

ensure_schema() -> None

Reconcile a writable ingest table to hold the locus identity plus the injected columns.

The identity is the variant_id key, genome, and the decomposed chr/pos/ref/alt locus that add_annotations and add_annotations_frame both write and the dedup reads filter on, clustered by [genome, chr, variant_id]. CREATE covers a fresh table; ADD COLUMN IF NOT EXISTS reconciles a pre-existing one by adding any missing columns (e.g. a table provisioned out of band). BigQuery cannot add a NOT NULL column, so identity columns added to an old table are nullable. Its earlier rows have no genome, so the dedup reports them unannotated, and because writes append, re-annotating them would leave a second row per variant. Run backfill_identity_columns once after this migration, before any dedup; ensure_schema never rewrites existing rows itself.

An identity-scoped source (see the class docstring) also adds a nullable _altar_annotation_identity STRING column. Added to an existing table, it is NULL on every earlier row, so those rows are neither served nor counted as annotated until backfill_annotation_identity adopts them. Without that backfill, the next deduplication re-annotates the whole lake. The clustering is unchanged.

backfill_identity_columns async

backfill_identity_columns() -> IdentityBackfill

Stamp the locus identity on rows written before the table carried it, as one transaction.

This is the explicit, one-time migration for a table that predates the identity columns (see ensure_schema, which must run first to add them; it never runs this). It assumes every legacy row, one with a NULL genome, belongs to genome_default's build, which is therefore required. Only a legacy row whose variant_id is canonical (_CANONICAL_VARIANT_ID_PATTERN, the RE2 form of VariantKey.require_canonical) is touched:

  • If its variant already has a row of genome_default's build (it was re-annotated before this ran), the legacy row is deleted.
  • Otherwise exactly one legacy row per variant_id is kept, stamped with genome_default and the chr/pos/ref/alt parsed from the variant_id, and the others are deleted. The survivor is the most recent by created_at when the table has that column, then the first by the row's JSON text, so a rerun on the same rows keeps the same one.

A legacy row with a non-canonical variant_id (1:9:A:T, chr6:-3:A:T, chr9:5:C:C) keeps its NULL genome: no canonical lookup can reach it, and stamping it would invent a locus. The work runs as one BigQuery multi-statement transaction (temp-table snapshot, DELETE, INSERT), so a concurrent write cannot interleave with it; stop ingest into the table while it runs and resume afterwards. A second run finds no canonical legacy rows and changes nothing. It returns how many rows were stamped and deleted, and as non_canonical how many legacy rows it left alone for their variant_id.

backfill_annotation_identity async

backfill_annotation_identity() -> IdentityBackfill

Adopt the rows written before this lake stored an annotation identity, as one transaction.

This is an explicit, reviewed one-time migration, not something Altar runs on its own. It asserts that every legacy row of genome_default's build, one whose _altar_annotation_identity is NULL, was produced by this source's configured identity (the same contract, data release and genome build). Run it only with the identity of the release that actually built those rows. Adopting them under a newer release would serve old values under the new identity, which is the error identity scoping exists to prevent.

The source must be identity-scoped (identity_scoped=True) and have a genome_default, and ensure_schema must have added the column. Rows with a NULL genome are not touched: run backfill_identity_columns first for those. Of the legacy rows of this build:

  • If its variant already has a row stamped with this identity (it was re-annotated before this ran), the legacy row is deleted.
  • Otherwise exactly one legacy row per variant_id is kept and stamped with the identity hash, and the others are deleted. The survivor is chosen as in backfill_identity_columns: the most recent by created_at when the table has that column, then the first by the row's JSON text.

A legacy row whose variant_id is not canonical (_canonical_variant_id_sql), such as chr9:5:C:C (REF equal to ALT) or 9:5:C:T, is neither stamped nor deleted. It stays unstamped, so the identity never serves or counts it, and the result reports how many there are as non_canonical so you can delete them. Once this has run, _altar_annotation_identity IS NULL AND genome = <genome_default> selects exactly those rows.

Legacy rows of another genome and rows stamped with any identity are left alone. Like backfill_identity_columns it runs as one BigQuery multi-statement transaction (temp-table snapshot, DELETE, INSERT); stop ingest into the table while it runs. A second run finds no canonical legacy rows of this build and changes nothing. It returns how many rows were stamped and deleted, and how many it left unstamped as non_canonical.

Cost: the snapshot reads every column of the legacy rows, and on-demand pricing bills the DELETE for every column of the whole table, so when most rows are legacy the transaction bills about three times the table's logical size and rewrites all of its storage. BigQuery re-clusters the rewritten rows in the background at no charge. The replaced storage stays in time travel and fail-safe for their configured windows, which a dataset on physical storage billing pays for.

add_annotations async

add_annotations(rows: Tabular) -> InsertResult

Append per-variant annotation rows, each a {"variant_id": ..., <column>: value, ...} mapping.

The write is append-only (WRITE_APPEND), so deduping is the caller's job (pre-flight via get_unannotated_variants). Like add_annotations_frame, every row is written with its genome and decomposed locus, so both write paths fill the identity columns the dedup reads filter on. variant_id must be canonical: a non-canonical one raises VariantIdentityError rather than being written where no read can find it. chr/pos/ref/alt are derived from it, and any the row carries must name the same variant. genome is the row's own, else genome_default (see __init__). An identity-scoped lake also stamps _altar_annotation_identity with its identity hash; a row that carries another hash raises AnnotationIdentityError. The table must already have that column (ensure_schema). A source that is not identity-scoped raises AnnotationIdentityError when its table has that column (see the class docstring).

add_annotations_frame

add_annotations_frame(frame: Any) -> InsertResult

Ingest a per-variant annotation frame columnar, the annotation counterpart of add_scores_frame.

It re-derives the variant_id (chr:pos:ref:alt) join key from the locus, overwriting any present-but-null passthrough id and rejecting a non-canonical locus with VariantIdentityError, stamps genome_default on a frame without a genome column (and rejects a genome column naming another build when genome_default is set), stamps an identity-scoped lake's identity hash (rejecting a frame whose _altar_annotation_identity names another), stamps created_at, and casts each declared scalar column to its physical dtype (nullable Int64, bool, or float64). repeated, struct, str, and json columns pass through untouched: BigQuery's load_table_from_dataframe (Parquet) maps a list-of-scalar to ARRAY<…> and a list-of-dict to ARRAY<STRUCT<…>>. It drops any af_* columns the caller's frame carries, since allele-frequency columns live in a separate source joined at materialize, so ALLOW_FIELD_ADDITION can't resurrect them. The write is append-only (WRITE_APPEND), so deduping is the caller's job (pre-flight via get_unannotated_variants), as in the score store. A source that is not identity-scoped raises AnnotationIdentityError when its table has _altar_annotation_identity (see the class docstring).

add_annotations_from_uri async

add_annotations_from_uri(
    uri: str | Sequence[str],
    *,
    expected_identity: AnnotationSourceIdentity
    | None = None,
    verify_identity: bool = True,
    staging_dataset: str | None = None,
    expiration: timedelta = _URI_LOAD_EXPIRATION,
) -> InsertResult

Load newline-delimited JSON annotation rows from uri into this lake, skipping variants it already has.

This is the warehouse-scale write for files an annotation job wrote to object storage, such as the region annotate command's JSONL. uri is any source URI, or list of URIs, a BigQuery load job accepts, such as gs://bucket/run-7/*.jsonl. Each row needs a canonical variant_id and may carry any of the declared columns. Fields that are not declared columns, such as the annotate JSONL's variant_type and af_* fields, are ignored.

expected_identity is the identity the rows were produced under, typically read from the job's identity sidecar with load_annotation_identity. When it is given, the load is refused with AnnotationIdentityError before anything is loaded if it differs from this source's identity, or if the source declares none. An identity-scoped source requires it, because it stamps its own identity on every row: without a check, files an outdated annotation image produced would be stored as the new release. verify_identity=False waives that requirement for a caller that has checked the files' identity itself.

The rows are loaded into a scratch table in staging_dataset (a project.dataset; the lake's own dataset by default) with the declared columns' types, so repeated and struct columns load natively. The scratch table expires after expiration as a backstop and is dropped when the call returns or fails. Then:

  • Every variant_id must be canonical (chr1:100:A:T, with different alleles). The annotate command copies the id from its input's fifth column and writes null without one; export_unannotated writes that column.
  • chr, pos, ref and alt fields the rows carry must name the same variant as the variant_id, as in add_annotations: pos exactly, the alleles ignoring case, and chr with or without its chr prefix (and MT for M). Rows without them are not checked.
  • On an identity-scoped lake, a row's own _altar_annotation_identity field must be empty or this source's identity hash.

A row that fails a check raises VariantIdentityError (or AnnotationIdentityError for the identity field) and nothing is inserted. Otherwise one INSERT … SELECT appends the rows, in the stored form of add_annotations:

  • genome is genome_default, which is required, and chr/pos/ref/alt are parsed from the variant_id.
  • An identity-scoped lake stamps _altar_annotation_identity with its identity hash.
  • A variant that already has a row of this build (and, when identity-scoped, of this identity) is skipped, so loading the same files twice adds nothing. The match is on variant_id, genome and the identity hash only, never on the stored chr, so a row written with another chr spelling still counts. That anti-join reads those three columns of every lake row of the build: on an 80-million-row lake, about 8.6 GB per load.
  • Rows of the same variant_id within the load collapse to one, the first by the row's JSON text.

It returns InsertResult(added=<rows inserted>, skipped=<loaded rows not inserted>); skipped counts both variants the lake already had and duplicate rows within the load. The lake must already have the columns (ensure_schema). Concurrent loads of the same variants into one lake can each insert a row, as with the other write paths; serialize them. A source that is not identity-scoped raises AnnotationIdentityError before loading anything when its table has _altar_annotation_identity (see the class docstring).

get_unannotated_variants async

get_unannotated_variants(
    variant_ids: Sequence[str],
    *,
    variants_relation: str | None = None,
    chromosomes: Sequence[str] | None = None,
) -> list[str]

Return the subset of variant_ids this lake has no annotation row for. This is the ingest dedup that lets an annotation job annotate only cache misses, the annotation counterpart of get_unscored_variants, differing only in the missing model_id (annotations are not per-model). It binds @variant_ids inline by default. Pass variants_relation (a backtick-quoted temp-table ref carrying variant_id) to prune at warehouse scale as materialize does, for a variant set too large to bind as an array param. On the relation path the full input lives only in the temp table, so the unannotated rows are read straight off a LEFT JOIN … IS NULL dedup and variant_ids may be empty. chromosomes then prunes the annotation table's chr cluster key; it is ignored on the inline path, whose small array binds directly. Both paths count only rows of genome_default's build, so it is required, and an identity-scoped lake counts only rows stamped with its identity hash, so a release bump reports every variant unannotated. It returns variant_ids, and the caller decomposes them back to loci (chr:pos:ref:alt) for the annotation-job input file.

Neither path ever returns a variant_id that is not canonical (VariantKey.require_canonical), such as chr9:100:C:C (REF equal to ALT), 9:100:C:T, or a NULL id in the relation. No write path stores a row for one, so reporting it would ask every job to annotate it again. export_unannotated leaves out the same ids and logs how many it left out.

export_unannotated async

export_unannotated(
    *,
    destination_uri: str,
    variants_relation: str,
    chromosomes: Sequence[str] | None = None,
) -> int

Export a job relation's unannotated variants to a headerless object-store TSV; return the count.

Each line is chr, pos, ref, alt, variant_id, the input the annotate command (altar.variants.annotation.annotate) reads: it copies the fifth column into each JSONL row's variant_id, which add_annotations_from_uri requires. chr, pos, ref and alt are split from the canonical variant_id, not read from the relation's own locus columns, so every line names the variant its id names and the loader's locus check cannot reject it. The relation needs only variant_id.

destination_uri needs one * wildcard, and BigQuery chooses how many files it writes; any of them may be empty. The annotate command reads one file, so annotate each shard (an empty one yields an empty JSONL) or concatenate them first, then load all the JSONL with one add_annotations_from_uri("gs://…/*.jsonl", expected_identity=…). Give every job a fresh prefix: overwrite=true replaces only the files this export writes, never stale files an earlier export left under the same wildcard, and an export of 0 variants writes nothing at all.

It counts and exports the same cache misses as get_unannotated_variants with variants_relation: only rows of genome_default's build, and, for an identity-scoped lake, of its identity hash, are annotated. Like it, it leaves out every unannotated variant whose variant_id is not canonical (VariantKey.require_canonical), such as chr9:100:C:C (REF equal to ALT), 9:100:C:T, or a NULL id, because the loader refuses a whole file for one such row. The return value counts only exported rows. When it left any out, it logs a warning with their count and an example, so you can delete them from the relation's source.

annotate async

annotate(variant_ids: Sequence[str]) -> AnnotationTable

Return {variant_id: {column: value}} for the requested variants this table has rows for. A variant with no row is absent, and the left-merge then yields {} for it (see merge_annotations). A declared scalar json column that the relation returns as JSON text, such as TO_JSON_STRING(...), is decoded to its JSON value, as SqliteAnnotationSource decodes its stored text. An identity-scoped lake reads only rows of its identity, through the same relation sql_join uses.

BigQueryAllelicRecordBackend

BigQueryAllelicRecordBackend(
    client: Any,
    table_id: str,
    *,
    key_columns: AllelicKeyColumns,
    filters: Mapping[str, str | int | bool] | None = None,
    batch_size: int = DEFAULT_FETCH_BATCH_SIZE,
)

Read source-native allelic records from one BigQuery table.

table_id is a project.dataset.table reference. key_columns names the table's chromosome, position, reference-allele, and alternate-allele columns. filters limits every read to rows whose columns equal fixed values. Use it when one table holds several genome builds or releases, for example {"genome": "hg38"}. Filter values are bound as query parameters.

Each requested key must describe the locus its variant_id names. A binding may spell the chromosome differently from the variant ID, for example 1 for chr1, but the position and alleles must match the ID's, as written or at their minimal representation. fetch raises ValueError for a key that does not. covered_variant_ids relies on the same rule to find candidate variants in SQL without calling the binding.

A fetch query reads the key and requested columns of the whole table, because BigQuery can skip storage only by clustering or partitioning. fetch therefore sends keys in batches of at most batch_size, one query per batch. Variants returned by the most recent covered_variant_ids call are answered from memory instead.

fetch async

fetch(query: AllelicRecordQuery) -> list[AllelicRecord]

Return every record matching query.keys, with only query.fields, in request order.

covered_variant_ids async

covered_variant_ids(variants_relation: str) -> list[str]

Return the variants in variants_relation that have at least one record in this table.

variants_relation is a SQL relation with a variant_id column in chr:pos:ref:alt form, such as a backtick-quoted table id. The match compares positions exactly and alleles ignoring case and surrounding spaces. It compares chromosomes after removing a chr prefix and leading zeros and treating MT as M. That comparison is looser than fetch, so it can include a variant fetch finds nothing for. It reads each ID as written: an ID that is not in minimal representation, which fetch would trim and find, is not covered.

The query also reads every column of the matched records and keeps them in memory. The next fetch for a covered variant is answered from those records, with the same exact key match a query would apply, and then releases them. If a covered variant's records are spelled differently from the requested key, for example 1 in the table and chr1 in the key, that fetch raises ValueError rather than return no records. A later call to this method replaces records that were never fetched. Point the backend at a table or view without unused wide columns to keep this read small.

LinkDatasetSummary dataclass

LinkDatasetSummary(
    rows: int, duplicates: int, files: tuple[Path, ...]
)

What write_link_dataset wrote: the rows, the exact duplicates it dropped, and the files. Provisional.

BuildInput dataclass

BuildInput(
    path: Path | None,
    name: str,
    uri: str | None = None,
    storage_options: Mapping[str, Any] = dict(),
)

One input VCF: a local file, a URI read through fsspec, or standard input when both path and uri are unset.

storage_options open a URI's filesystem; they never enter equality or the text of the input.

name instance-attribute

name: str

The file name, or the name standard input stands for, which names the output and is checked by pattern.

is_stdin property

is_stdin: bool

Return whether this input is standard input.

stem property

stem: str

Return the output name: name without its .vcf or .tsv, .gz, and .bgz suffixes.

label property

label: str

Return how messages name this input; a URI is named without its query, which may hold a signature.

BuildOutput

BuildOutput(
    output: str | PathLike[str],
    *,
    partition_column: str,
    label: str,
    metadata: Mapping[str, str],
    connection: Connection | None = None,
    unique_key: Sequence[str] = (VARIANT_ID_COLUMN,),
)

Where a builder writes its dataset: a local directory, or a URI that each input's files are uploaded to.

output is a local path, a file:// URI, or a URI such as gs://bucket/gnomad-4.1.1-genomes, opened with connection.storage_options(output) or with the provider's ambient credentials when there is no connection. Every file the build writes records metadata, its partition directories are <partition_column>=<value>, and label names the build in messages. unique_key names the columns no two rows of a partition may share (check_unique_keys): variant_id alone for one row per allele, or variant_id and a column such as a transcript for a dataset with several records per allele. Provisional: the shape is expected to change.

Use open around the build. Each input is built by build_input, which reads a remote input again from the start when its stream fails or is cut short, at most attempts times. A local output is written in place, as build_output writes it. For a URI output, each input is built in a local scratch directory (under TMPDIR), its keys are checked there (check_unique_keys), and each finished file is uploaded under a temporary name and committed, then deleted locally, so the scratch holds one input's files at a time. A reader of the output never sees a partial file.

A URI output records its progress under _altar_build/ until the build finishes, so running the same command again after a crash resumes it: the inputs whose files were all committed are skipped (resume), and the others are built again. An instance is picklable, so worker processes build and upload their own inputs.

resumed instance-attribute

resumed: tuple[str, ...] = ()

The names of the inputs resume skipped, because an unfinished build had already committed them.

remote property

remote: bool

Return whether the output is a URI that files are uploaded to, rather than a local directory.

open

open(*, overwrite: bool) -> Iterator[Self]

Prepare the output for the body of the with, which builds every input and checks the build.

A local output behaves as build_output: it must be empty or absent, or hold a previous build that overwrite deletes first, and a failed body deletes everything the build wrote.

A URI output must be empty or absent, or hold an unfinished build of the same release (one whose _altar_build/build.json records the same metadata, apart from the builder version), which is resumed. With overwrite, a previous build there, finished or not, is deleted first. Either way, anything a build does not write (files outside <partition_column>=*/*.parquet and _altar_build/) is refused, and so is, without overwrite, a finished build or an unfinished build of another release. A failed body keeps the files already committed, so the same command resumes the build; when the body succeeds, _altar_build/ is deleted and the output is a finished build.

resume

resume(inputs: Sequence[BuildInput]) -> list[BuildInput]

Return the inputs still to build, in order; set resumed to the names of those skipped.

An input is skipped when the unfinished build open resumes committed every file it wrote. Each one skipped is logged. A local output, or a fresh one, skips nothing.

build_input

build_input(
    source: BuildInput,
    attempt: Callable[[Path], Built],
    *,
    attempts: int = DEFAULT_INPUT_ATTEMPTS,
) -> Built

Build one input with attempt, which writes its files under the directory it is given and lists them.

attempt must delete what it wrote when it fails. When it fails on a remote input with a cut-short or failed stream (TruncatedInputError, a stream that ended inside a gzip member, or a transient network error), it is called again on an empty directory after a pause, up to attempts calls in all; after that an OSError names the input and the last error. Any other error, and any error on a local file or standard input, is raised at once.

For a local output, attempt writes into the output and its result is returned as it is. For a URI output, it writes into a fresh scratch directory; the keys in each of its partition files are checked to be unique, each file is uploaded and committed (each upload tried up to attempts times) and deleted locally, the input is recorded as committed, and the result's files are replaced by the URIs of the uploaded files. Must be called inside open, in this process or in a worker process given this instance.

check_unique_keys

check_unique_keys() -> None

Reject a unique_key written twice in a partition of the output, across inputs or by trimming.

A local output is checked in full (check_unique_keys). A URI output's files were each checked before they were uploaded, so only partitions that hold several files, from several inputs, are checked again, by reading their key columns from the output; a build of one input per chromosome reads nothing. A failure leaves the build unfinished: fix the inputs and rebuild with overwrite.

OpenInput

OpenInput(source: BuildInput)

A text view of one input, whose compression is read from its first bytes rather than its name.

Use it as a context manager and read either text, line by line, or header_lines and then tsv_batches, which parse the rest of the input in bulk; the two views share one stream, so a builder uses one or the other. Standard input is left open on exit. A URI is streamed through its fsspec filesystem in blocks, never downloaded whole. After reading to the end, check_complete rejects a bgzipped input that was cut short.

header_lines

header_lines() -> Iterator[str]

Yield the input's lines one at a time, read without reading ahead, so tsv_batches starts after them.

Stop iterating at the last header line; the lines the caller took are not parsed again. Provisional, like BuildOutput.

tsv_batches

tsv_batches(
    column_names: Sequence[str],
    *,
    block_size: int = _BATCH_BYTES,
) -> Iterator[RecordBatch]

Parse the rest of the input as tab-separated text into record batches of string columns.

Each batch holds the lines of about block_size bytes, parsed by PyArrow's CSV reader on its own threads, so memory stays bounded however large the input. Every line must have exactly the named columns; there is no quoting, an empty line is skipped, and an empty field is the empty string, never null. A line with another number of columns raises ValueError naming the input and the line. A stream that fails or ends early raises the error the stream raised, such as the EOFError of a cut-short gzip member, which BuildOutput.build_input retries for a remote input. Provisional, like BuildOutput.

convert_batches

convert_batches(
    column_names: Sequence[str],
    convert: Callable[[RecordBatch, int], T],
    *,
    workers: int | None = None,
    block_size: int = _BATCH_BYTES,
) -> Iterator[T]

Parse the rest of the input with tsv_batches and yield convert(batch, first_line) for each, in order.

first_line numbers the batch's first row among the data rows, from 1, so convert can name a line in an error. The conversions run on workers threads (by default up to 8), which overlap because Arrow's compute releases the GIL, while this generator reads the next batches; at most twice workers batches are in flight, so memory stays bounded. convert must not change shared state: do what depends on the order of the input, such as RecordOrder and PartitionedWriter, with the results as they are yielded. An error raised by convert is raised when its batch's turn comes, and the batches after it are not converted. Provisional, like BuildOutput.

check_complete

check_complete() -> None

Raise TruncatedInputError if a bgzipped input ended without bgzip's end-of-file block.

ParquetReleaseMetadata dataclass

ParquetReleaseMetadata(
    prefix: str,
    distribution: str,
    label: str,
    command: str,
    legacy_prefixes: tuple[str, ...] = (),
    layout: str | None = None,
)

How one binding records its data release in the key-value metadata of every Parquet file it builds.

prefix namespaces the keys, as in altar_popfreq.; the fields under it are the binding's own, such as a version and a dataset, or RELEASE_METADATA_FIELD and GENOME_BUILD_METADATA_FIELD. write adds BUILDER_VERSION_METADATA_FIELD, the installed version of distribution. label names the data in messages and command is the builder a rejected file should be rebuilt with. legacy_prefixes are prefixes earlier builders wrote: read takes a field under the current prefix first, then under each legacy prefix in turn.

layout names the file layout the binding's source reads, for a binding whose builder changed its columns or partitions after datasets were built with the old ones. write records it as LAYOUT_METADATA_FIELD, and read_dataset and check reject a file that records this binding's metadata but another layout, or none, telling the reader to rebuild it, before they compare any release field: a file of the old layout would otherwise be read wrong. A file with none of the binding's metadata is refused as before. A binding whose files never changed layout leaves it None, and nothing is written or checked.

write

write(fields: Mapping[str, str]) -> dict[str, str]

Return the metadata to write into a file: fields under prefix, layout, and the builder's version.

read

read(
    path: str | PathLike[str],
    *,
    storage_options: Mapping[str, Any] | None = None,
) -> dict[str, str]

Return the fields a Parquet file records, without their prefix; the current prefix wins over a legacy one.

path is a local path or a URI, read as ParquetAllelicRecordBackend reads its dataset, with storage_options passed to a remote filesystem. Only the file's footer is read.

read_dataset

read_dataset(
    dataset: str | PathLike[str],
    *,
    required: Sequence[str],
    storage_options: Mapping[str, Any] | None = None,
) -> dict[str, str]

Return the fields the first Parquet file under dataset records, which must include required.

dataset is a file or a directory searched recursively, as a local path or a URI with storage_options for its filesystem (see read); the first file in sorted order is read, and a binding's source then holds the dataset to the release it names with check. For a remote dataset, the next check of the same dataset with the same options reuses this read instead of listing the dataset and reading the footer again, so a source that takes its release from the dataset and then checks it reads one footer. Raises FileNotFoundError when there is no Parquet file and ValueError, naming the missing fields and the builder to rebuild with, when the file does not record every required field or has another layout.

check

check(
    dataset: str | PathLike[str],
    expected: Mapping[str, str],
    *,
    storage_options: Mapping[str, Any] | None = None,
    all_files: bool = False,
) -> None

Raise ValueError if the first Parquet file under dataset, or with all_files any, is not expected.

A file fails when it records none of expected's fields, or other values for them, or another layout than this metadata's. dataset is a file or a directory searched recursively, as a local path or a URI with storage_options for its filesystem (see read); a dataset with no Parquet file is left to the backend. Only footers are read.

By default only the first file in sorted order is checked, which is what a source does when it opens a dataset: a release is identified at the release level, so a later file built from another release is not caught there, and opening a remote dataset of many files costs one footer read rather than one per file. When the last read_dataset of the same remote dataset has not been checked yet, its read is reused and nothing is read. all_files reads every file's footer, as altar pull does to verify a copy it makes.

PartitionedWriter

PartitionedWriter(
    output: Path,
    stem: str,
    *,
    schema: Schema,
    partition_column: str,
    row_group_rows: int,
    metadata: Mapping[str, str],
    parquet_options: Mapping[str, Any] | None = None,
)

Write one input's rows to <output>/<partition_column>=<chromosome>/<stem>.parquet, one file per chromosome.

Each row holds every column of schema; the file records metadata in its schema and is written in row groups of row_group_rows with zstd compression. A file is created when its chromosome's first row arrives and is written under a .partial name until commit renames it, so a reader never sees a file in progress.

parquet_options are further keyword arguments to pyarrow.parquet.ParquetWriter, such as use_dictionary=["chr", "score"] and column_encoding={"pos": "DELTA_BINARY_PACKED"} for a column of increasing positions; they may override compression. Without them a file is written exactly as before the option existed. Provisional, like append_batch.

append

append(chromosome: str, row: Mapping[str, object]) -> None

Buffer one row of chromosome's file.

append_batch

append_batch(
    chromosome: str, batch: Table | RecordBatch
) -> None

Buffer a table or record batch of chromosome's rows, as append would buffer each row in turn.

The batch holds every column of schema, in any order and with any other columns ignored, each of a type that casts to the schema's; a non-nullable column must hold no null. Rows keep their order, and the file's row groups are the same as if the rows had come one at a time, so a builder may mix the two. An empty batch creates no file. Provisional, like BuildOutput.

commit

commit() -> list[Path]

Finish every file and return their paths, in the order their chromosomes first appeared.

abort

abort() -> None

Delete every partial file.

RecordOrder

RecordOrder(label: str, *, kind: str = 'VCF')

Require sorted input: each chromosome contiguous, positions non-decreasing within it, no key twice at one.

Call advance with each record's chromosome and position, then claim with each allele key it yields. A key is (position, reference, alternate) after trimming, so a trimmed key may move past the record's position; a key repeated across positions is caught by check_unique_keys over the finished output. kind names the input format in messages, such as "TSV".

advance

advance(chromosome: str, position: int) -> None

Move to the next record, rejecting one out of order.

advance_batch

advance_batch(
    chromosome: str, positions: Array | Sequence[int]
) -> None

Move past a run of records on one chromosome at once, as advance would for each position in turn.

Rejects the same out-of-order record, with the same message, as advance called once per position. Provisional, like BuildOutput.

claim

claim(chromosome: str, key: tuple[int, str, str]) -> None

Record an allele key at the current position, rejecting one already written there.

TruncatedInputError

Bases: ValueError

A bgzipped input ended without bgzip's end-of-file block: a download or stream was cut short.

A remote input that ends this way is read again (BuildOutput.build_input); a local file is rejected.

PositionScoreAnnotationSource

Bases: AnnotationSource

Annotate variants with a base-wise score track, such as phyloP, from a position score dataset. Provisional.

The source implements contract, which declares exactly one float column, and reads a dataset in the POSITION_SCORE_METADATA layout built for that contract (its footer's track). Build it with from_dataset.

indels chooses which variants are scored. "skip" (the default) scores SNVs only, by the score at their base. "max" scores every variant by the greatest score over the bases variant_positions gives it: an SNV's base, a deletion's deleted bases, the two bases an insertion falls between, or another change's REF span; unscored bases are ignored. SNVs score the same in both modes. Because "max" scores variants "skip" leaves out, its identity's release is the dataset's release plus +indels-max.

A variant is read at its minimal representation (minimal_variant_key), and its row is keyed by the ID as passed. A variant with no scored base, or on a contig the dataset has no partition for (an alt contig, say), has no row. annotate reads all of a call's variants together, each needed row group once. prioritize_predicate, evaluated over the contract's column, makes the source gate prioritized; without one it only annotates. name is the logical source name, by default the contract's column name. A dataset is on hg38 or hg19 (supported_genome_builds), and the source's identity names the one its footer records.

release property

release: str

The identity's release: the dataset's release, plus +indels-max when indels are scored.

from_dataset classmethod

from_dataset(
    dataset: str | PathLike[str],
    *,
    contract: AnnotationContract,
    indels: str = "skip",
    genome_build: str | None = None,
    storage_options: Mapping[str, Any] | None = None,
    prioritize_predicate: Predicate | None = None,
    name: str | None = None,
) -> Self

Read the position score dataset at dataset, a local path or a URI such as gs://... or hf://....

The dataset is read in place, with storage_options passed to its fsspec filesystem. Its release and genome build come from the first file's footer (POSITION_SCORE_METADATA). ValueError is raised for a contract that is not exactly one float column, an indels other than "skip" or "max", a footer of another layout or none, a footer whose track is not contract, and a genome_build other than the footer's.

OwnedDatasetCopy dataclass

OwnedDatasetCopy(
    dataset: str | PathLike[str],
    storage_options: Mapping[str, Any] | None,
)

An owned Parquet copy a run spec names with dataset (PublishedDataset.select_copy).

Provisional, like PublishedDataset.

Parameters:

Name Type Description Default
dataset str | PathLike[str]

The copy's path or URI, as the spec gave it.

required
storage_options Mapping[str, Any] | None

The fsspec options that read it, from the spec or its connection; None when neither gives any, so the provider's ambient credentials apply.

required

OwnedTableCopy dataclass

OwnedTableCopy(
    table: str, release: str, connection: Connection
)

An owned BigQuery table a run spec names with table (PublishedDataset.select_copy).

Provisional, like PublishedDataset.

Parameters:

Name Type Description Default
table str

The table, project.dataset.table, as the spec gave it; the binding's table source checks it.

required
release str

The release label the table is read as: the spec's release, or the published release. It is passed on unparsed, so the binding's own release type validates it.

required
connection Connection

The connection whose Connection.bigquery_client reads the table.

required

PublishedCopy dataclass

PublishedCopy(
    location: PublishedLocation,
    storage_options: dict[str, Any],
)

The published copy a run spec reads by default: one of a PublishedDataset's locations (select_copy).

Provisional, like PublishedDataset.

Parameters:

Name Type Description Default
location PublishedLocation

The location to read, the primary unless the options named another.

required
storage_options dict[str, Any]

The fsspec options that read it: the spec's own, a connection's, or anonymous ones.

required

uri property

uri: str

Return the location's URI, the dataset a source opens.

PublishedDataset dataclass

PublishedDataset(
    locations: tuple[PublishedLocation, ...],
    identity: AnnotationSourceIdentity,
    release_metadata: ParquetReleaseMetadata | None = None,
    release_fields: Mapping[str, str] = dict(),
    cluster_by: tuple[str, ...] = (VARIANT_ID_COLUMN,),
)

One release of a binding's data, published as the binding's Parquet layout at one or more public locations.

Provisional: see the module docstring.

Parameters:

Name Type Description Default
locations tuple[PublishedLocation, ...]

Where the release's files are hosted, in order of preference: the first is the primary, which a reader uses unless it names another, and the rest are mirrors holding the same files, such as a Hugging Face copy of a Cloud Storage dataset. Names are unique. A location never enters identity: every location serves the same release.

required
identity AnnotationSourceIdentity

The contract, data release, and genome build of the published data: the identity a source serving it declares, whichever location or copy it is read from.

required
release_metadata ParquetReleaseMetadata | None

How the binding's builder records the release in each file's footer, the ParquetReleaseMetadata its source checks. With release_fields, it lets check_release hold a copy to this release; a dataset that declares neither cannot be copied with a check.

None
release_fields Mapping[str, str]

The footer fields, without their prefix, that every file of this release records, such as {"version": "4.1.1", "dataset": "genomes", "genome_build": "hg38"}. It must hold GENOME_BUILD_METADATA_FIELD with the identity's genome build, and RELEASE_METADATA_FIELD, when present, must be the identity's release.

dict()
cluster_by tuple[str, ...]

The columns a BigQuery table loaded from the dataset is clustered on, in order, by pull_to_bigquery (altar pull --bigquery) unless its caller names others: the columns the binding's table reads filter on. By default variant_id, which every allele-keyed builder writes; a link dataset, read by interval overlap, declares ("chromosome", "start"). At most four, as BigQuery allows; () for none. Provisional.

(VARIANT_ID_COLUMN,)

uri property

uri: str

Return the primary location's URI, the one a reader uses unless it names another location.

release property

release: str

Return the data release label the dataset serves, such as "4.1.1-genomes".

genome_build property

genome_build: str

Return the assembly of the dataset's variant coordinates, such as "hg38".

storage_options property

storage_options: dict[str, Any]

Return the fsspec options that read the primary location anonymously (PublishedLocation).

location

location(name: str | None = None) -> PublishedLocation

Return the location named name, or the primary (the first) when name is None.

Raises ValueError for a name the dataset does not list, naming the ones it does.

read_storage_options

read_storage_options(
    connection: Connection | None = None,
    *,
    location: str | None = None,
) -> dict[str, Any]

Return the fsspec options to read the location named location, the primary by default.

They are connection's for that location's URI, or anonymous ones; see PublishedLocation.read_storage_options. An unknown location raises ValueError.

check_release

check_release(
    dataset: str | PathLike[str],
    *,
    storage_options: Mapping[str, Any] | None = None,
    all_files: bool = True,
) -> None

Raise ValueError unless every Parquet file under dataset records this release and genome build.

dataset is a copy of this dataset, or one of its locations itself, as a local path or a URI read with storage_options; only each file's footer is read, with release_metadata against release_fields. Every file is checked, as verifying a copy needs; all_files=False checks only the first file in sorted order, as a source does when it opens a dataset (ParquetReleaseMetadata.check). A dataset that declares no release_metadata raises ValueError, since its copies cannot be checked.

select_copy

select_copy(
    label: str,
    *,
    dataset: str | PathLike[str] | None = None,
    table: str | None = None,
    release: str | None = None,
    storage_options: Mapping[str, Any] | None = None,
    connection: Connection | None = None,
    location: str | None = None,
) -> PublishedCopy | OwnedDatasetCopy | OwnedTableCopy

Return the copy of this dataset's data that a source's run spec options name, and how to reach it.

A binding that publishes its data calls this from AnnotationSource.from_options with the options it was given, so every such binding reads the same three modes by the same rules:

  • With neither dataset nor table, a PublishedCopy: the location named location (the primary by default), read with storage_options when given, or else with connection's storage options for its URI, or else anonymously (PublishedLocation.read_storage_options).
  • With dataset, an OwnedDatasetCopy: an owned Parquet copy at that path or URI, read with connection's storage options for it, or with storage_options; with neither, the storage options are None and the provider's ambient credentials apply.
  • With table, an OwnedTableCopy: an owned BigQuery table, read through connection, which it needs. A table holds no Parquet metadata, so release names the release it holds, and defaults to this dataset's release, which a table loaded from it holds. The binding parses and checks the label.

Options that do not name one readable copy raise ValueError, with messages that begin with label, the data's name, such as "gnomAD": both dataset and table; location with either, since it picks where the published copy is read from; storage_options with a connection (which supplies them) or a table (which needs none); release without a table, since Parquet files record their own release; a table without a connection; and an unknown location. Nothing is read: the binding opens the copy, checks its release, and builds its source. No option chosen here enters the source's identity.

PublishedLocation dataclass

PublishedLocation(name: str, uri: str)

One place a published dataset's files are hosted, under a short name a user picks it by.

Provisional: see the module docstring.

Parameters:

Name Type Description Default
name str

A short lowercase name, such as "gcs" or "hf": letters, digits, _ and -, starting with a letter.

required
uri str

The dataset's directory there, as a URI that an fsspec filesystem reads in place, such as gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes or hf://datasets/altar-genomics/gnomad-4.1.1-genomes. The scheme needs its provider's extra: altar[gcs] for gs://, altar[hf] for hf://. A partitioned dataset is found by listing its directory, which plain HTTPS cannot do on Cloud Storage, so a dataset hosted there is named by its gs:// URI.

required

scheme property

scheme: str

Return the URI's scheme, lowercased, such as "gs" or "hf".

storage_options property

storage_options: dict[str, Any]

Return the fsspec options that read the location anonymously: {"token": "anon"} for gs://.

read_storage_options

read_storage_options(
    connection: Connection | None = None,
) -> dict[str, Any]

Return the fsspec options to read the location: connection's for uri, or anonymous ones.

A connection reads the same data with other credentials, such as a project billed for requester-pays reads; it never changes which data is read. A connection that cannot reach the URI's scheme raises CapabilityNotSupported.

BigQueryLoadSummary dataclass

BigQueryLoadSummary(
    table: str,
    source_uri: str,
    job_id: str | None,
    rows: int | None,
    write_disposition: str,
    clustering_fields: tuple[str, ...],
    identity: AnnotationSourceIdentity,
    labels: Mapping[str, str] = dict(),
)

What pull_to_bigquery loaded: the table, the source files, the load job, and the release they hold.

Provisional, like PublishedDataset.

labels class-attribute instance-attribute

labels: Mapping[str, str] = field(default_factory=dict)

The release labels written on the table (bigquery_release_labels); empty if labelling it failed.

to_dict

to_dict() -> dict[str, Any]

Return the summary as a JSON-ready mapping.

PullProgress dataclass

PullProgress(
    stage: Literal["start", "copied"],
    files_total: int,
    bytes_total: int,
    files_present: int = 0,
    bytes_present: int = 0,
    files_done: int = 0,
    bytes_done: int = 0,
    path: str | None = None,
    file_bytes: int = 0,
)

How far pull_dataset or pull_to_bigquery has got, as passed to their progress callback.

Provisional, like PublishedDataset.

stage is "start" once the source files are listed and sized, before anything is copied or loaded, and "copied" after each file pull_dataset commits; pull_to_bigquery reports "start" only, since its one load job is atomic. Sizes are in bytes.

files_total instance-attribute

files_total: int

The number of Parquet files in the published dataset.

bytes_total instance-attribute

bytes_total: int

Their total size.

files_present class-attribute instance-attribute

files_present: int = 0

Of those, the files already at the destination with the source's size, which a resumed pull skips.

bytes_present class-attribute instance-attribute

bytes_present: int = 0

The total size of the files already present.

files_done class-attribute instance-attribute

files_done: int = 0

The files in place so far, those already present included.

bytes_done class-attribute instance-attribute

bytes_done: int = 0

The bytes in place so far, those already present included.

path class-attribute instance-attribute

path: str | None = None

The file just copied, relative to the dataset's root; None at the start.

file_bytes class-attribute instance-attribute

file_bytes: int = 0

The size of the file just copied; 0 at the start.

PullSummary dataclass

PullSummary(
    destination: str,
    files: int,
    bytes: int,
    identity: AnnotationSourceIdentity,
    source_uri: str,
    skipped: int = 0,
)

What pull_dataset copied: the number of files and bytes, where from and to, and the release they hold.

Provisional, like PublishedDataset.

files and bytes count what this call copied. A resumed pull also counts the files it found already complete and skipped (skipped), so files + skipped is the number of files in the copy.

source_uri instance-attribute

source_uri: str

The URI of the published location the files were copied from.

skipped class-attribute instance-attribute

skipped: int = 0

The files already at the destination with the source's size, which this call did not copy again.

to_dict

to_dict() -> dict[str, Any]

Return the summary as a JSON-ready mapping.

JoinSpec dataclass

JoinSpec(
    alias: str,
    relation: str,
    join_key: str,
    columns: list[str],
)

One LEFT JOIN the store adds to its materialize query to read an annotation source in SQL.

Adding this join lets the store read the source within its single query instead of calling annotate() and merging in Python. For each spec the store builds:

`LEFT JOIN <relation> AS <alias> ON <alias>.<join_key> = <base_alias>.<join_key>`

and adds <alias>.<column> for each entry in columns to the materialized row.

relation is whatever goes between LEFT JOIN and AS. It is a backtick-quoted table reference for a per-variant table, or a parenthesized subquery such as (SELECT variant_id, … FROM t GROUP BY variant_id) when the source first collapses several per-transcript rows into one row per variant (AlphaMissense and REVEL do this inside the source, not the store). A source that cannot fuse returns None from sql_join, and the store reads it through annotate() instead.

SqlAnnotationSource

Bases: AnnotationSource

An AnnotationSource that shares a SQL backend with the store and can fuse its read into the store's materialize query.

A source subclasses this to signal that the store may read it inside one SQL query rather than through annotate() and a Python merge. A SQL store finds these sources with isinstance(source, SqlAnnotationSource). A plain AnnotationSource (Parquet, in-process, or a different backend) does not subclass it, so the store reads that source through annotate(). Subclassing is how a source opts in; the base AnnotationSource contract has no SQL methods.

sql_join abstractmethod

sql_join(ctx: SqlJoinContext) -> JoinSpec | None

Return a JoinSpec the store adds to its materialize query, or None if this source cannot fuse for the given ctx. When it returns None, the store reads this source through annotate() instead.

SqlJoinContext dataclass

SqlJoinContext(
    base_alias: str,
    join_key: str = "variant_id",
    variant_ids_param: str = "variant_ids",
    variants_relation: str | None = None,
)

The information the store passes each source at materialize time so it can build a join (sql_join).

It holds only what the store's materialize step currently needs.

  • base_alias — the store's base relation, the scores side, which each source's join attaches to as the left side.
  • join_key — the column both sides join on (variant_id).
  • variant_ids_param — the name of the array query parameter the store binds to the job's variant ids (@variant_ids). A source whose relation is an aggregating subquery, where AlphaMissense or REVEL collapse several per-transcript rows into one, uses this to restrict the large reference table to the job's variants before aggregating, with an INNER JOIN variants scan. The outer LEFT JOIN cannot do this, because it runs after the GROUP BY and so cannot prune the group's input.
  • variants_relation — an optional SQL relation (a table id or subquery) that yields the job's variant_ids. When set, {variant_filter} prunes with variant_id IN (SELECT variant_id FROM <relation>) instead of the inline UNNEST(@variant_ids) parameter. A job with millions of variants uses this to scope the query without exceeding BigQuery's request-size limit: it uploads a temp table and passes the reference here. The default, None, uses the inline-parameter prune, which suits small jobs and the emulator.

SqliteAnnotationSource

SqliteAnnotationSource(
    path: str = ":memory:",
    columns: Sequence[AnnotationColumn] = (),
    *,
    name: str | None = None,
    prioritize_predicate: Predicate | None = None,
    identity: AnnotationSourceIdentity | None = None,
    identity_scoped: bool = False,
)

Bases: AnnotationSource

Your own per-variant values, from a SQLite file. Build it in code with the columns the file holds.

It uses one table, variant_annotations, keyed by variant_id, with one column per AnnotationColumn. It acts as a prioritizing source when you pass a prioritize_predicate, and a passive one otherwise. That choice is configuration, not a separate subclass.

Unlike the read-only base class, this source can also be written to. Use add_annotations to load your own reference data so it feeds into materialize, and get_unannotated_variants to find the variants it has no row for yet.

identity= alone is metadata. With identity_scoped=True, the file holds the annotations of exactly one AnnotationSourceIdentity, recorded in its _altar_annotation_identity table. A scoped source records its identity the first time it opens a file with no annotation rows. Opening a file that records a different identity raises AnnotationIdentityError, so a data release bump uses a new file rather than serving the old release's rows. A file that already holds rows but records no identity was written before identities were stored: a scoped source opens it, but refuses to read or write it until backfill_annotation_identity() adopts its rows. Each operation re-reads the recorded identity, so an adoption by another instance is seen. An unscoped source reads any file, but cannot write to one that records an identity.

backfill_annotation_identity async

backfill_annotation_identity() -> IdentityBackfill

Adopt a legacy file's rows as this source's identity, and return how many rows were adopted.

This is an explicit, reviewed one-time migration. It records the configured identity in a file that holds annotation rows but no identity, asserting that the declared data release produced every row in it. It needs identity_scoped=True. It returns IdentityBackfill(stamped=<rows adopted>, deleted=0): the primary key already allows one row per variant, so nothing is deleted. A file that already records this identity is left unchanged and stamped is 0, so a second call is a no-op; a file recording another identity cannot be opened at all.

add_annotations async

add_annotations(rows: Tabular) -> InsertResult

Insert per-variant annotation rows. Each row is a mapping like {"variant_id": ..., <column>: value, ...}. Rows use INSERT OR IGNORE on the variant_id primary key, so a variant that already has a row is skipped rather than overwritten.

For an identity-scoped source, the rows belong to the file's identity, and a row whose _altar_annotation_identity names another identity hash raises AnnotationIdentityError. An unscoped source cannot write to a file that records an identity.

annotate async

annotate(variant_ids: Sequence[str]) -> AnnotationTable

Return {variant_id: {column: value}} for the requested variants that this source has rows for. A variant with no row is left out of the result; the merge step then treats it as an empty annotation (see merge_annotations).

get_unannotated_variants async

get_unannotated_variants(variant_ids: Tabular) -> list[str]

Return the variant_ids this file has no annotation row for, in input order.

For an identity-scoped source the file holds one identity (see the class docstring), so every stored row is a row of this source's identity.

close

close() -> None

Close the source's SQLite connection; a second call does nothing.

Close a file-backed source when you are done with it, or open it in a with block, which closes it on exit.

VariantGeneLinkAnnotationSource

VariantGeneLinkAnnotationSource(
    store: VariantGeneLinkStore,
    dataset: VariantGeneLinkDatasetRef,
    *,
    release: str,
    genome_build: str = "hg38",
    filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
    prioritize_predicate: Predicate | None = None,
    name: str | None = None,
    column_prefix: str | None = None,
)

Bases: AnnotationSource

The genes each variant may regulate, from variant-gene links you import, such as ENCODE-rE2G.

It annotates variants with the org.kundajelab.altar.annotation.variant-gene-links columns from a link store. A run spec declares it over a SQLite link store.

store is any VariantGeneLinkStore and dataset the published dataset in it to read; the in-memory and SQLite stores, or a host's own, serve alike. release is the data release label of that dataset, which the store does not record itself; it must change whenever the stored evidence changes. genome_build is spelled as in score identity, "hg38" or "hg19" (supported_genome_builds), and selects the assembly the variant IDs are on; a dataset that holds another assembly raises GenomeBuildMismatchError from annotate. filters narrow every variant's query, for example to one biosample; because they change the values, the identity's release becomes <release>+<fingerprint of the filters> when any is set. prioritize_predicate, evaluated over the contract's columns such as link_top_score, makes the source gate prioritized; without one it only annotates. name is the logical source name recorded in prioritized_source_names.

column_prefix lets two link datasets annotate one materialization, for example ENCODE-rE2G next to Open Targets E2G. With a prefix such as "re2g", every declared column is re2g_<column> (re2g_link_top_score, re2g_link_genes; the fields inside link_genes keep their names), labels name the prefix, and the default name is re2g_variant_gene_links, so the sources' names and columns stay distinct. A predicate then reads the prefixed columns. Without a prefix the columns are the contract's own. The prefix renames columns and does not change their values, so identity() is the contract's either way. A model's dependency on the contract reads the contract's column names, which only an unprefixed source declares.

Over a store, each requested variant is queried at its own locus through the store's public query contract, every page of links is read, and the result is projected with project_variant_gene_links. A variant's locus is the REF span of its minimal representation (minimal_variant_key), and its row is keyed by the ID as passed. The locus includes the anchor base an indel's ID carries, so a deletion links to an element that ends at its anchor base, although phyloP scores only the deleted bases. A variant on a chromosome the link contract does not name, such as an unplaced contig, has no row. Store failures raise VariantGeneLinkSourceError; they are never turned into an empty row.

One locus query per variant suits tests, notebooks and cohorts of up to some tens of thousands of variants. For larger cohorts, write the links as a Parquet link dataset (write_link_dataset) and read it with from_dataset, which reads many variants per query, locally or where the dataset is hosted, and projects the same rows through the same function; its read failures raise VariantGeneLinkSourceError too. A link dataset loaded into BigQuery is read with from_bigquery_table, one range-join query per chunk of variants.

release property

release: str

The identity's release: the dataset's release, plus a fingerprint of the filters when any is set.

from_dataset classmethod

from_dataset(
    dataset: str | PathLike[str],
    *,
    storage_options: Mapping[str, Any] | None = None,
    genome_build: str | None = None,
    filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
    prioritize_predicate: Predicate | None = None,
    name: str | None = None,
    column_prefix: str | None = None,
) -> Self

Read the Parquet link dataset write_link_dataset wrote at dataset, without a link store. Provisional.

dataset is a local path or a URI such as gs://... or hf://..., read in place, with storage_options passed to its fsspec filesystem (see ParquetAllelicRecordBackend). The release, the genome build and the VariantGeneLinkDatasetRef come from the metadata the writer records in every file (LINK_DATASET_METADATA), read from the first file; a file that records none raises ValueError. genome_build, when given, must be the build the metadata records, or GenomeBuildMismatchError is raised. The other options are the constructor's; with filters the identity's release is the recorded release plus the filters' fingerprint, as for a store.

annotate reads every requested variant's links by row group rather than one query per variant: the files' footers, read once, say which row groups can hold a link overlapping each variant, and each such row group is read once however many variants it serves, adjacent ones together; a remote dataset is read with one ranged request per run of adjacent row groups. Its rows are projected through project_variant_gene_links, as a store's are, so the two modes return the same rows for the same links.

from_bigquery_table classmethod

from_bigquery_table(
    client: Any,
    table_id: str,
    *,
    dataset: VariantGeneLinkDatasetRef,
    release: str,
    genome_build: str = "hg38",
    filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
    prioritize_predicate: Predicate | None = None,
    name: str | None = None,
    column_prefix: str | None = None,
) -> Self

Read a link dataset loaded into the BigQuery table table_id through client. Provisional.

table_id is a project.dataset.table holding a link dataset's columns, as altar pull --bigquery loads one. A table holds no Parquet metadata, so dataset, release and genome_build say what it holds, and the table's release labels are checked against them with one metadata call, no query (check_bigquery_release): a table labelled with another release or build raises ValueError, and one with no labels is read with a warning. The other options are the constructor's; with filters the identity's release is release plus the filters' fingerprint, and the labels are checked against release itself.

annotate runs one query per TABLE_QUERY_LOCI variants: their loci are bound as array query parameters and range-joined against the table with the filters, and the links are projected through project_variant_gene_links, so the rows are those of the Parquet and store modes. The source does not own client; whoever made it closes it. A query BigQuery refuses raises VariantGeneLinkSourceError.

from_sqlite classmethod

from_sqlite(
    path: str | PathLike[str],
    dataset: VariantGeneLinkDatasetRef,
    *,
    release: str,
    genome_build: str = "hg38",
    filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
    prioritize_predicate: Predicate | None = None,
    name: str | None = None,
    column_prefix: str | None = None,
) -> Self

Read dataset from the existing SqliteVariantGeneLinkStore file at path.

The file must already hold a published generation of dataset (see SqliteVariantGeneLinkStore.open_published), so a mistyped path or dataset fails here instead of annotating nothing. The source owns the store's connection, and close closes it.

from_options classmethod

from_options(**options: Any) -> Self

Build the source over a SQLite link store from a run spec's options.

The options are sqlite_path, dataset as a {namespace, name, version} mapping, and release, plus optionally genome_build, column_prefix, name, and filters, a mapping of VariantGeneLinkFilters fields whose identifier filters are lists of strings. A predicate cannot be spelled in a spec, so a spec-declared source only annotates; build the source in code to prioritize with it::

[sources.re2g]
type = "variant_gene_links"

[sources.re2g.options]
sqlite_path = "links.db"
dataset = { namespace = "encode", name = "re2g", version = "ENCFF229GDZ" }
release = "ENCFF229GDZ"
column_prefix = "re2g"
filters = { context_names = ["right kidney"] }

close

close() -> None

Close what the source opened: the store from_sqlite opened, or the reader from_dataset opened.

A second call does nothing. A store passed to the constructor and a client passed to from_bigquery_table stay open, because whoever created them closes them.

VariantGeneLinkFilters dataclass

VariantGeneLinkFilters(
    context_ids: tuple[str, ...] = (),
    context_names: tuple[str, ...] = (),
    source_ids: tuple[str, ...] = (),
    gene_ids: tuple[str, ...] = (),
    methods: tuple[str, ...] = (),
    data_releases: tuple[str, ...] = (),
    minimum_score: float | None = None,
    maximum_score: float | None = None,
)

Relation-level filters applied to every variant's link query, with VariantGeneLinkQuery semantics.

Context IDs and names select their union; the other identifier tuples and the inclusive score bounds are intersections. Values are stripped and deduplicated. The filters change which links a variant has, so VariantGeneLinkAnnotationSource folds them into its identity's release.

BiosampleContext dataclass

BiosampleContext(
    name: str,
    id: str | None = None,
    namespace: str | None = None,
)

The cell, tissue, or other biosample in which a link was measured or predicted.

GenomeBuild

Bases: StrEnum

Genome assemblies supported by the normalized link contract.

GenomeBuildMismatchError

Bases: VariantGeneLinkQueryError

The query and source use different genome assemblies.

GenomicInterval dataclass

GenomicInterval(
    chromosome: str,
    start: int,
    end: int,
    genome_build: GenomeBuild,
    coordinate_system: CoordinateSystem = "0-based-half-open",
)

A normalized 0-based, half-open genomic interval [start, end).

overlaps

overlaps(other: GenomicInterval) -> bool

Return whether two same-build intervals overlap.

InvalidCursorError

Bases: VariantGeneLinkQueryError

A pagination cursor is malformed or belongs to a different query.

LinkDistances dataclass

LinkDistances(
    element_to_gene_bp: float | None = None,
    variant_to_gene_bp: float | None = None,
    variant_to_element_bp: float | None = None,
    interpretation: str | None = None,
)

Distances retained when a source provides them, in base pairs.

LinkProvenance dataclass

LinkProvenance(
    source_record_id: str,
    source_dataset: str,
    source_url: str | None = None,
    study_id: str | None = None,
    publication_ids: tuple[str, ...] = (),
    metadata: Mapping[str, Any] = dict(),
)

Evidence identifiers and source-specific metadata retained without flattening.

LinkScore dataclass

LinkScore(
    value: float | None,
    direction: ScoreDirection,
    interpretation: str,
    threshold: float | None = None,
    threshold_rule: ThresholdRule | None = None,
    threshold_applied: bool | None = None,
    threshold_interpretation: str | None = None,
)

A source score without conflating a missing value with numeric zero.

RegulatoryElement dataclass

RegulatoryElement(
    id: str,
    interval: GenomicInterval,
    element_type: str | None = None,
)

Identity and genomic extent of one regulatory element.

ScoreDirection

Bases: StrEnum

How to interpret increasing values of a source's link score.

TargetGene dataclass

TargetGene(
    id: str,
    namespace: str,
    namespace_version: str | None = None,
    symbol: str | None = None,
)

A target gene identifier with an explicit namespace and optional release/symbol.

VariantGeneLink(
    record_id: str,
    locus: VariantLocus,
    regulatory_element: RegulatoryElement | None,
    target_gene: TargetGene,
    context: BiosampleContext | None,
    source_id: str,
    source_name: str,
    method: str,
    data_release: str,
    score: LinkScore,
    distances: LinkDistances,
    provenance: LinkProvenance,
)

One loss-minimizing variant/locus-to-gene evidence record.

regulatory_element is present for mediated evidence such as enhancer-to-gene links. It is absent for sources that directly associate a locus or allele with a gene. Consumers that specifically project an enhancer graph must reject direct records instead of inventing a regulatory element.

VariantGeneLinkDataError

Bases: VariantGeneLinkSourceError

The source returned malformed, unstable, or ambiguously duplicated data.

VariantGeneLinkError

Bases: Exception

Base exception for variant-to-gene link operations.

VariantGeneLinkPage dataclass

VariantGeneLinkPage(
    links: tuple[VariantGeneLink, ...],
    next_cursor: str | None = None,
)

A bounded page of links; an empty tuple with no cursor is a successful empty result.

VariantGeneLinkQuery dataclass

VariantGeneLinkQuery(
    locus: VariantLocus,
    context_ids: tuple[str, ...] = (),
    context_names: tuple[str, ...] = (),
    source_ids: tuple[str, ...] = (),
    gene_ids: tuple[str, ...] = (),
    methods: tuple[str, ...] = (),
    data_releases: tuple[str, ...] = (),
    minimum_score: float | None = None,
    maximum_score: float | None = None,
    page_size: int = 100,
    cursor: str | None = None,
)

One interval-overlap query plus relation-level filters and a bounded page size.

Context IDs and names are alternative selectors (their union). Other non-empty identifier tuples are intersections, as are the inclusive score bounds. No score cutoff is applied when both bounds are absent.

VariantGeneLinkQueryError

Bases: VariantGeneLinkError, ValueError

The normalized query is invalid for the selected source.

VariantGeneLinkSource

Bases: ABC

Queryable one-to-many variant/locus-to-gene evidence.

A valid query with no matching links returns VariantGeneLinkPage(links=(), next_cursor=None). Network, storage, malformed-source-data, and pagination failures raise VariantGeneLinkSourceError (or a subclass) and must never be translated into an empty page.

query_links(
    query: VariantGeneLinkQuery,
) -> VariantGeneLinkPage

Return one stable, bounded page of links for query.

stream_links(
    query: VariantGeneLinkQuery,
) -> AsyncIterator[VariantGeneLink]

Stream all pages without retaining the full relation in memory.

VariantGeneLinkSourceError

Bases: VariantGeneLinkError

A source could not answer a valid query because of network, storage, or source-data failure.

VariantLocus dataclass

VariantLocus(
    interval: GenomicInterval,
    reference_allele: str | None = None,
    alternate_allele: str | None = None,
)

A normalized query locus, optionally carrying normalized alleles.

interval always uses 0-based, half-open coordinates. from_vcf is the explicit conversion from a 1-based VCF/Open Targets variant position; the reference allele determines the reference span.

from_vcf classmethod

from_vcf(
    *,
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
    genome_build: GenomeBuild,
) -> VariantLocus

Convert a 1-based VCF locus into the contract's 0-based, half-open interval.

The interval is the REF span, so an indel's locus includes the VCF anchor base before the inserted or deleted bases.

InMemoryVariantGeneLinkStore

InMemoryVariantGeneLinkStore()

Deterministic reference store used by examples and Cartesian conformance tests.

LinkRecordWriteResult dataclass

LinkRecordWriteResult(added: int, skipped: int)

Counts returned by an idempotent generation write.

SqliteVariantGeneLinkStore

SqliteVariantGeneLinkStore(path: str | Path)

Portable SQLite store for canonical evidence with indexed overlap and relation filters.

close

close() -> None

Close the store's SQLite connection; a second call does nothing.

open_published classmethod

open_published(
    path: str | PathLike[str],
    dataset: VariantGeneLinkDatasetRef,
) -> Self

Open an existing store file to read dataset, which must have at least one published generation.

The constructor creates a missing file and its schema, which suits an importer. A reader configured from a path, such as a run spec's link annotation source, uses this instead, so a mistyped path or dataset fails here rather than reading as a dataset without links. Raises FileNotFoundError for a missing file and ValueError when the file has no published generation of dataset.

StoredVariantGeneLinkSource

StoredVariantGeneLinkSource(
    store: VariantGeneLinkStore,
    dataset: VariantGeneLinkDatasetRef,
    *,
    name: str,
    binding_id: str,
    binding_version: str,
    failure_label: str = "variant-gene link store",
)

Bases: VariantGeneLinkSource

Reusable public source over a generic canonical evidence store.

decode_record

decode_record(
    record: VariantGeneEvidenceRecord, locus: VariantLocus
) -> VariantGeneLink

Project a canonical record into a result; bindings may restore typed source metadata.

VariantGeneEvidenceRecord dataclass

VariantGeneEvidenceRecord(
    record_id: str,
    evidence_interval: GenomicInterval,
    regulatory_element: RegulatoryElement | None,
    target_gene: TargetGene,
    context: BiosampleContext | None,
    source_id: str,
    source_name: str,
    method: str,
    data_release: str,
    score: LinkScore,
    distances: LinkDistances,
    provenance: LinkProvenance,
    metadata: Mapping[str, JsonValue] = dict(),
)

One durable, query-independent locus/element-to-gene evidence record.

to_link(locus: VariantLocus) -> VariantGeneLink

Attach a caller's overlapping locus and return the public query result.

VariantGeneLinkDatasetRef dataclass

VariantGeneLinkDatasetRef(
    namespace: str, name: str, version: str
)

Stable identity for one independently versioned collection of canonical link evidence.

key property

key: str

Return a deterministic store key without assigning semantics to separators in components.

VariantGeneLinkStore

Bases: Protocol

Persist canonical evidence independently of its scientific source format.

Records are written into immutable generations and become queryable only after publish_generation. This lets importers checkpoint bounded batches without exposing partial releases.

write_records async

write_records(
    dataset: VariantGeneLinkDatasetRef,
    generation_id: str,
    records: Sequence[VariantGeneEvidenceRecord],
) -> LinkRecordWriteResult

Idempotently stage a bounded batch in generation_id.

publish_generation async

publish_generation(
    dataset: VariantGeneLinkDatasetRef, generation_id: str
) -> None

Atomically make all records in a staged generation visible to readers.

generation_size async

generation_size(
    dataset: VariantGeneLinkDatasetRef, generation_id: str
) -> int

Return the number of unique staged or published records in a generation.

available_builds async

available_builds(
    dataset: VariantGeneLinkDatasetRef,
) -> Sequence[GenomeBuild]

Return genome builds represented by published generations.

fetch_page async

Return published records ordered strictly by record_id.

VariantGeneLinkStoreQuery dataclass

VariantGeneLinkStoreQuery(
    interval: GenomicInterval,
    context_ids: tuple[str, ...] = (),
    context_names: tuple[str, ...] = (),
    source_ids: tuple[str, ...] = (),
    gene_ids: tuple[str, ...] = (),
    methods: tuple[str, ...] = (),
    data_releases: tuple[str, ...] = (),
    minimum_score: float | None = None,
    maximum_score: float | None = None,
    after_record_id: str | None = None,
    limit: int = 101,
)

Backend-neutral indexed request derived from a public link query.

from_query classmethod

Translate the public query while decoding its opaque cursor once in the source layer.

canonical_chromosome

canonical_chromosome(chromosome: str) -> str

Return the canonical chr-prefixed spelling used in portable keys.

Primary chromosomes accept common aliases (1/chr1/CHR01 and M/MT/chrM) in any case. Other reference contigs keep their exact spelling after an optional case-insensitive chr prefix is normalized, so CHRUn_KI270302v1 becomes chrUn_KI270302v1 but chrun_KI270302v1 stays as written. Non-ASCII text, whitespace inside the name, and characters outside the VCF contig-name set (including :) raise VariantIdentityError. This is a naming rule only; it does not claim that contigs from different assemblies are interchangeable.

canonical_variant_id

canonical_variant_id(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
) -> str

Return the canonical text key for decomposed one-based locus fields.

minimal_representation

minimal_representation(
    position: int,
    reference_allele: str,
    alternate_allele: str,
) -> tuple[int, str, str]

Trim the bases that REF and ALT share, keeping at least one base in each.

The shared suffix is removed first, then the shared prefix, and position advances by one for each prefix base removed, so the result names the same change at the same locus: (10, "CAT", "CT") becomes (10, "CA", "C") and (10, "TCA", "TGA") becomes (11, "C", "G"). Trimming the suffix first leaves an indel anchored on the base before it, the VCF convention. A SNV, and alleles that share no end base (A/AT, AT/A), come back unchanged. Alleles are spelled as the key spells them: trimmed of ASCII whitespace and uppercased.

The alleles must have a key: each is [ACGTN]+ and they differ, like VariantKey. Symbolic (<DEL>), spanning-deletion *, breakend and missing alleles raise VariantIdentityError rather than being trimmed as if their characters were bases, and so do equal alleles, which would otherwise collapse to one shared base that is not a variant. The position is not validated beyond being an int, so a one-based or zero-based position stays in its own convention.

Trimming is a pure function of the alleles. Left alignment, which shifts an indel through a repeat using the reference sequence, is a separate step that ingestion applies around this one.

allelic_key

allelic_key(variant_id: str) -> AllelicKey

Return the backend key for a canonical chr:pos:ref:alt variant ID.

The key's variant_id is the ID as passed. Its other fields are the parts of minimal_variant_key: a chr-prefixed chromosome, the one-based position, and the trimmed reference and alternate alleles. A malformed or non-canonical ID raises VariantIdentityError (a ValueError). A collection that spells chromosomes another way, such as 1 rather than chr1, can rewrite the key's chromosome with dataclasses.replace; the key keeps the caller's variant_id either way.

Provisional: the helper is expected to change.

minimal_variant_key

minimal_variant_key(variant_id: str) -> VariantKey

Return the key an annotation lookup reads for a canonical chr:pos:ref:alt variant ID.

The ID is checked with VariantKey.require_canonical, which raises VariantIdentityError (a ValueError) for a malformed or non-canonical one. Its alleles are then trimmed to their minimal representation (minimal_representation), so chr17:43124027:ACTC:AC reads as chr17:43124027:ACT:A. Trimming needs no reference. Left alignment does, so an indel written at the right end of a repeat keeps its position.

allelic_key, PositionScoreAnnotationSource and VariantGeneLinkAnnotationSource look variants up through this function and key their rows by the ID the caller passed.

Provisional: the helper is expected to change.

applicable_annotation_sources

applicable_annotation_sources(
    models: Sequence[ScoredModel],
    sources: Sequence[AnnotationSource],
) -> tuple[AnnotationSource, ...]

Return the sources that may annotate the models' variants, in configuration order.

The variants' assembly is the genome build the models' run identities record. A source whose identity() declares another assembly is dropped. A source that declares no identity serves the default assembly (DEFAULT_GENOME_BUILD, hg38): it is kept for hg38 variants and dropped for variants on any other assembly, whether or not Altar knows its label. With no model recording a genome build, every source is kept. Models scored on different assemblies raise AnnotationGenomeBuildError when any source is configured, because no single assembly then decides which sources apply.

Every store's materialize and export call this and join only the returned sources. It never raises for a dropped source; resolve_annotation_dependencies decides whether a drop is a configuration error.

resolve_annotation_dependencies

resolve_annotation_dependencies(
    models: Sequence[ScoredModel],
    sources: Sequence[AnnotationSource],
    *,
    contracts: Sequence[AnnotationContract] = (),
) -> tuple[ResolvedAnnotationDependency, ...]

Check that sources satisfy every annotation dependency of models, and return how.

For each AnnotationDependency in each model's manifest.prioritization:

  1. If a source's identity().source_id is the dependency's source_id, that source must implement a compatible contract version (same major version, and a version no older than the dependency's) and declare every dependency column. Each of those columns must have the logical type (dtype, repeated and nested fields; labels may differ) that every known contract with the dependency's source_id and major version gives it. Otherwise AnnotationDependencyError is raised.
  2. If no source declares that contract but every dependency column is declared by some source, the dependency resolves by column name and an UndeclaredAnnotationContractWarning is emitted.
  3. Otherwise AnnotationDependencyError is raised naming the contract and the missing columns.

Only the sources applicable_annotation_sources keeps take part. A dependency they do not satisfy is:

  • unavailable, when source classes implement the dependency's contract and none of them serves the variants' assembly. It resolves with the missing columns in unavailable, and one AnnotationUnavailableWarning per contract names the models;
  • AnnotationGenomeBuildError otherwise, when a dropped source would have supplied it (it declares the contract or supplies the missing columns): a source on the variants' assembly could exist, so the dropped one is a configuration error;
  • AnnotationDependencyError otherwise, as in step 3.

The classes are the healthy altar.annotation_sources entries, plus the classes of the configured sources, whose annotation_contract() names the contract. A class serves an assembly when its supported_genome_builds contains it or is None. A contract no class implements could be served on any assembly. Whether a mismatch warns or raises therefore depends on which source packages are installed. A dropped source no dependency needed is named in one AnnotationUnavailableWarning. The UCSC and GRC spellings of one assembly, such as hg38 and GRCh38, match. With no model recording a genome build, every source that declares an identity must declare the same assembly; otherwise AnnotationGenomeBuildError is raised.

The known contracts are the ones Altar publishes, such as REGION_ANNOTATION_CONTRACT, plus contracts. A dependency on a contract the resolver does not know, such as a binding's or a host's own, gets the name check only. Pass that contract in contracts to check its column types too, or run altar.testing.assert_implements_contract to check a source against its whole contract.

Store materialize methods, and the BigQuery exports that compute prioritized, call this once per call, without contracts. A host can also call it up front, with its own contracts, to check a configuration before submitting work.

unavailable_annotation_columns

unavailable_annotation_columns(
    resolved: Iterable[ResolvedAnnotationDependency],
) -> frozenset[str]

Return every column the resolved dependencies mark unavailable; a store evaluates each as missing (null).

fetch_annotations async

fetch_annotations(
    sources: Sequence[AnnotationSource],
    variant_ids: Sequence[str],
) -> dict[str, dict[str, Any]]

Fetch each source's annotations for variant_ids and merge them with merge_annotations.

This builds the annotation input to materialize_variant from the configured annotation sources. Sources are fetched concurrently and merged in declaration order. Their declared column names must be globally unique, and the sources that declare an identity() must declare one genome build, as in materialization. Either problem raises before any source is called: a collision SchemaCollisionError, a second build AnnotationGenomeBuildError. Each annotate must return the portable variant_id -> {column: value} mapping.

Each ID must be canonically spelled (VariantKey.require_canonical); a non-canonical one raises VariantIdentityError. The result is keyed by the IDs as passed. The shipped allelic, position and variant-gene link sources look each ID up at its minimal representation (minimal_variant_key), so chr17:43124027:ACTC:AC finds the row stored for chr17:43124027:ACT:A. A writable lake, such as SqliteAnnotationSource, stores and reads rows under the ID as written. Left alignment is the caller's job: no source left-aligns, so left-align indels against the reference first (altar.variants.left_align), or an indel written at the right end of a repeat finds no row.

get_annotation_source_registry

get_annotation_source_registry() -> PluginRegistry[
    type[AnnotationSource]
]

Return the shared registry of annotation sources (the altar.annotation_sources group).

load_annotation_identity

load_annotation_identity(
    sidecar: str
    | bytes
    | PathLike[str]
    | Mapping[str, Any],
) -> AnnotationSourceIdentity

Read the AnnotationSourceIdentity recorded in an annotation identity sidecar.

The region annotate command writes the sidecar as <out>.identity.json next to its JSONL rows. It is a JSON object with "format": 1, the identity's four fields under "identity", and their "identity_hash"; other keys, such as columns, provenance and row_count, are informational and ignored here.

sidecar is the sidecar's JSON text (str or bytes), its already-parsed object, or a local file path (os.PathLike, such as a pathlib.Path; a plain str is always JSON text). Altar never fetches the sidecar from object storage: a host reads the bytes with its own client and passes them here, then passes the result as expected_identity to BigQueryAnnotationSource.add_annotations_from_uri.

Raises AnnotationIdentityError when the sidecar is not a JSON object, has another format, lacks identity, or records an identity_hash that the identity does not produce, and ManifestError when the identity fields themselves are invalid.

merge_annotations

merge_annotations(
    variant_ids: Sequence[str],
    tables: Sequence[AnnotationTable],
    *,
    source_names: Sequence[str] | None = None,
) -> dict[str, dict[str, Any]]

Left-merge several sources' annotation tables into one mapping per variant.

The left side is variant_ids, the variants being scored. Every id gets an entry, even one that no source annotated, which yields {}. Each table is the portable variant_id -> {column: value} shape a source returns from annotate. Two sources may not supply the same column: doing so is ambiguous even when their declared dtypes agree, because their values may differ. A row a source returns for a variant not in variant_ids is dropped.

This is the portable join that builds the annotation argument to materialize_variant. The BigQuery store instead fuses its annotation reads into one query through SqlAnnotationSource.sql_join.

staged_annotation_source async

staged_annotation_source(
    source: AnnotationSource,
    client: Any,
    *,
    staging_dataset: str,
    variants_relation: str | None = None,
    variant_ids: Sequence[str] | None = None,
    coverage: BigQueryAllelicRecordBackend | None = None,
    batch_size: int = DEFAULT_STAGE_BATCH_SIZE,
    expiration: timedelta | None = DEFAULT_STAGE_EXPIRATION,
) -> AsyncIterator[BigQueryAnnotationSource]

Stage source's annotations for one job in a scratch table and yield a SQL source bound to it.

source is a source without SQL of its own. The yielded source has the same name, columns, identity, and prioritization rule, and reads the scratch table. Its joins are limited to each store query's variants.

Give the job's variants as exactly one of variant_ids or variants_relation, a SQL relation with a variant_id column such as a backtick-quoted table id. With variants_relation and a coverage backend, one query finds the variants that have records and reads those records. Only those variants are then annotated. Without coverage, every variant id in the relation is read into Python and annotated.

staging_dataset is the project.dataset that holds scratch tables. Each table expires after expiration as a backstop. The table is dropped when the block exits, including on error or cancellation. With expiration=None a table left behind by a crashed process is never removed.

Raises ValueError if source returns a variant it was not asked for, and SchemaCollisionError if it returns undeclared columns.

link_dataset_release_fields(
    *,
    release: str,
    genome_build: str,
    dataset: VariantGeneLinkDatasetRef,
) -> dict[str, str]

Return the footer fields that identify a link dataset's release, without their prefix. Provisional.

genome_build is Altar's spelling (hg38 or hg19), as PublishedDataset requires; the rows keep the link contract's spelling. dataset is recorded as its namespace, name and version.

link_dataset_row(
    record: VariantGeneEvidenceRecord,
) -> dict[str, Any]

Return the link dataset row of one stored evidence record: its interval and its link_genes record.

Provisional, like write_link_dataset.

write_link_dataset(
    rows: Iterable[Mapping[str, Any] | RecordBatch | Table],
    output: str | PathLike[str],
    *,
    release: str,
    genome_build: str,
    dataset: VariantGeneLinkDatasetRef,
    overwrite: bool = False,
    temp_directory: str | PathLike[str] | None = None,
) -> LinkDatasetSummary

Write link rows as a link dataset in the local directory output. Provisional.

rows yields link rows as mappings (link_dataset_row makes one from an evidence record) or as Arrow record batches or tables, in any mix and any order; each has exactly the layout's columns: chromosome, start, end, genome_build and the LINK_GENE_FIELDS. A chromosome is in the link contract's spelling (7, not chr7) and genome_build is the contract's name for genome_build (GRCh38 for hg38). The rows are consumed once, in batches, so a build of tens of millions of links streams.

The rows are staged in a DuckDB database under temp_directory (by default the system's), which also takes the sort's spill, so memory stays bounded. A row repeated exactly is written once and counted in duplicates; a record_id on two different rows raises ValueError, as does a dataset with no rows or a row that breaks the layout. The rows are then written sorted by chromosome, start, end and record_id, one file per chromosome under chromosome=<value>/, in row groups of 32,000 rows, each file recording link_dataset_release_fields through LINK_DATASET_METADATA and written under a .partial name until it is complete.

output must be empty or absent; with overwrite, a previous link dataset there is replaced. A failed write leaves nothing behind. Write locally, then copy the directory to a bucket.

add_release_argument

add_release_argument(
    parser: ArgumentParser,
    *,
    help: str,
    deprecated: Sequence[str] = (),
    required: bool = True,
) -> None

Add --release, the upstream data release a builder reads, to parser; its value is args.release.

help is the option's help text. Each spelling in deprecated, such as "--version", sets the same value and is left out of --help. Using one prints warning: <spelling> is deprecated; use --release to standard error. Giving two different values is a usage error.

build_inputs

build_inputs(
    input_paths: Sequence[str | PathLike[str]],
    stdin_name: str | None = None,
    *,
    connection: Connection | None = None,
) -> list[BuildInput]

Resolve a builder's inputs: local files, URIs, and standard input once as STDIN, named stdin_name.

A URI such as gs://bucket/chr1.vcf.bgz or https://host/chr1.vcf.gz is read in place through fsspec, opened with connection.storage_options(uri), or with the provider's ambient credentials when there is no connection; its name is the last segment of its path. A file:// URI is a local file. A missing fsspec or provider raises an ImportError that names the Altar extra to install.

Raises ValueError when there is no input, standard input is listed twice, stdin_name is given without STDIN or is not a bare file name, or two inputs would write the same output name; and FileNotFoundError when a file or a URI does not exist.

build_output

build_output(
    output: Path,
    *,
    overwrite: bool,
    partition_column: str,
    label: str,
) -> Iterator[Path]

Make output an empty directory for the body of the with, and delete what the body wrote if it fails.

The directory must be empty or absent. With overwrite, a previous build there, <partition_column>=* directories of Parquet files and nothing else, is deleted first; anything else is refused, naming label as the build that would not have written it. If the body raises, the output is removed when this call created it, and emptied otherwise.

canonical_chromosome_runs

canonical_chromosome_runs(
    chromosomes: Array | ChunkedArray,
) -> list[tuple[str, int, int]]

Split a batch of records into runs on one chromosome: (canonical chromosome, start, stop) row ranges.

Each spelling is canonicalized once with canonical_chromosome, so 1 and chr1 side by side are one run, and an invalid name raises its VariantIdentityError. Sorted input has one run per chromosome in a batch; feed each run's positions to RecordOrder.advance_batch. Provisional, like BuildOutput.

check_unique_keys

check_unique_keys(
    output: Path,
    *,
    unique_key: Sequence[str] = (VARIANT_ID_COLUMN,),
) -> None

Reject a unique_key written twice in a partition of output, across inputs or by trimming.

unique_key is variant_id alone for a dataset of one row per allele, or variant_id and the columns that tell an allele's records apart, such as a transcript. Keys never span chromosomes, so each <partition_column>=* directory is checked on its own.

minimal_representations

minimal_representations(
    positions: Array, references: Array, alternates: Array
) -> tuple[Array, Array, Array]

Trim a batch of alleles as minimal_representation trims each: return their positions, REFs and ALTs.

The alleles must already have a key ([ACGTN]+, uppercase, REF and ALT different). Only alleles whose REF and ALT both have more than one base can share a base to trim, so the others are returned as they are and only those are trimmed, one at a time. Provisional, like BuildOutput.

require_rows

require_rows(
    rows: int, *, label: str, read: str, hint: str
) -> None

Reject a build that wrote no rows: an empty dataset is a wrong input, not an annotation of nothing.

read says what the build consumed, such as "3 records read, 3 skipped", and hint what to check.

storage_option_connection

storage_option_connection(
    pairs: Sequence[str],
) -> Connection | None

Return a connection with the storage options a builder's KEY=VALUE arguments give, or None for none.

A value that reads as JSON other than a string takes that type, so anon=true is a boolean and target_options={"auto_mkdir": true} a mapping; any other value is a string. Credentials never come from a command line: a key that would carry one, such as secret or token (other than gcsfs's token=anon and its other credential modes), is refused, and the provider's ambient credentials are used instead. Provisional, like BuildOutput.

position_score_release_fields

position_score_release_fields(
    *,
    contract: AnnotationContract,
    release: str,
    genome_build: str,
) -> dict[str, str]

Return the footer fields that identify a position score dataset's release, without their prefix. Provisional.

contract is the track's contract, which must declare exactly one float column; its source_id is recorded as the track, so a dataset built for one contract is never read under another. genome_build is hg38 or hg19; an alias such as GRCh38 is recorded in Altar's spelling.

bigquery_release_labels

bigquery_release_labels(
    identity: AnnotationSourceIdentity,
) -> dict[str, str]

Return the BigQuery table labels that record the release identity names.

Provisional, like PublishedDataset.

Three labels record the contract's source_id (altar_source), the release (altar_release) and the genome_build (altar_genome_build). A label value may hold only lowercase letters, digits, _ and -, at most 63 of them, so each value is the identity field lowercased, with every other character, such as ., replaced by _: release 4.1.1-genomes is 4_1_1-genomes, and source org.kundajelab.altar.annotation.gnomad is org_kundajelab_altar_annotation_gnomad. A longer value keeps its first 54 characters, then - and the first 8 hex digits of the SHA-256 of the original text.

The encoding is not reversible, so labels are compared, never decoded: check_bigquery_release encodes the identity a source expects and compares the result. The table's description names the release in full.

check_bigquery_release

check_bigquery_release(
    client: Any,
    table: str,
    identity: AnnotationSourceIdentity,
) -> bool

Check that the BigQuery table table holds the release identity names, as far as its labels say.

Provisional, like PublishedDataset.

Reads the table's labels with one metadata request (client.get_table): no query runs and no bytes are billed. A table whose Altar labels (bigquery_release_labels) name another source, release or genome build raises ValueError, and one with all three matching returns True. A table with no Altar label, such as one loaded by hand rather than by pull_to_bigquery, is accepted with a UserWarning naming the release it is assumed to hold, and False is returned. A client without get_table, such as a test double, cannot be checked and also returns False. An error looking the table up, such as a missing table, propagates.

The labels say which release was loaded, not that the rows were never edited since; the check is a cheap guard against reading one release under another's identity.

pull_dataset

pull_dataset(
    published: PublishedDataset,
    destination: str | PathLike[str],
    *,
    location: str | None = None,
    source_connection: Connection | None = None,
    destination_connection: Connection | None = None,
    overwrite: bool = False,
    progress: Callable[[PullProgress], None] | None = None,
) -> PullSummary

Copy every Parquet file of published to destination, keeping their paths, and check the copy's release.

Provisional, like PublishedDataset.

The files are read from the published location named location, such as "hf" for a Hugging Face mirror, or from the primary location by default (PublishedDataset.location); every location holds the same files, so the copy is the same whichever is read. An unknown name raises ValueError naming the dataset's locations.

destination is a local directory or a URI such as gs://my-bucket/gnomad-4.1.1-genomes; the files keep their paths relative to the location's URI, so the copy has the same <partition>=<chromosome>/ layout and a source reads it as an owned dataset. The published files are read anonymously, or with the storage options of source_connection, such as one that bills a requester-pays read. The copy is written with the storage options of destination_connection, or with the provider's ambient credentials when none is given.

Files are streamed one at a time, in chunks, never held whole in memory. Each is written under a temporary name and committed when complete: renamed into place on a local disk, finalized on an object store, which makes an object visible only once its upload completes. A reader of the destination therefore never sees a partial file, and a failed copy leaves at most the files committed before it.

A pull resumes: the destination may be absent, empty, or hold files of this dataset, such as those an interrupted pull committed. A file whose path is already there with the source's size is skipped, and every other file is copied; a temporary file a killed pull left beside one of the dataset's files is removed. A destination holding any other file is refused, unless overwrite is set; then every file is copied again, replacing files of the same path, and nothing else is deleted. A Parquet file that is not part of the dataset is refused either way, before anything is copied, because a source would read it too.

After copying, every Parquet file under destination must be one of the dataset's, and every one, copied or skipped, must record the published release and genome build (PublishedDataset.check_release); otherwise ValueError is raised, naming what to remove or rebuild. A skipped file of the same size but another release therefore fails the pull rather than passing as this one.

progress, when given, is called with a PullProgress once the files are listed and sized ("start", which counts the files already present) and after each file is copied ("copied").

Raises FileExistsError for a destination holding other files without overwrite, FileNotFoundError when the published dataset lists no Parquet file, and ValueError when published declares no release metadata, so the copy could not be checked. Returns the number of files and bytes copied, the number of files skipped, and the URI they came from.

pull_to_bigquery

pull_to_bigquery(
    published: PublishedDataset,
    table: str,
    *,
    connection: Connection,
    location: str | None = None,
    source_uri: str | None = None,
    source_connection: Connection | None = None,
    overwrite: bool = False,
    cluster_by: Sequence[str] | None = None,
    progress: Callable[[PullProgress], None] | None = None,
) -> BigQueryLoadSummary

Load the Parquet files of published into the owned BigQuery table table with one load job.

Provisional, like PublishedDataset.

table is a project.dataset.table reference in a dataset that exists. The load job runs and is billed through connection.bigquery_client(), as the connection's identity. BigQuery loads only from Cloud Storage, so the files are read from the published location named location (the primary by default), which must then be a gs:// URI, or from source_uri, a gs:// copy made with pull_dataset, such as one in the table's region or one pulled from a mirror that is not on Cloud Storage. location and source_uri are exclusive. BigQuery reads the files itself, and the destination dataset's location must be one that can load from the bucket's.

Before loading, the release metadata of every source file is checked against the published release, read with source_connection (or anonymously) for the published URI and with connection for source_uri. A table holds no Parquet metadata, so after the load the release is written as table labels (bigquery_release_labels), which a source reading the table checks (check_bigquery_release), and the table's description names it in full. Labelling is one metadata update; if it fails, the loaded table is kept, a warning says so, and the summary's labels is empty.

The load takes <uri>/*.parquet (or the one file a .parquet URI names), so every file under the dataset is loaded, whatever its partition directory. Hive partitioning is not used: every file already holds its partition column (chr in the builders' layouts), and BigQuery refuses partition keys that are also columns of the files. The table is clustered on cluster_by, by default the dataset's own (PublishedDataset.cluster_by): variant_id, which every allele-keyed builder writes, or chromosome, start for a link dataset; pass () for none. The load infers Parquet lists, so a repeated column becomes an ARRAY, and a list of records an ARRAY<STRUCT<...>>, as a binding's SQL relation reads it; without list inference, BigQuery would keep Parquet's encoding, STRUCT<list ARRAY<STRUCT<element ...>>>.

The table must not exist or be empty (WRITE_EMPTY); overwrite replaces its rows (WRITE_TRUNCATE). A load job is atomic, so an interrupted pull loads nothing and is simply run again. progress, when given, is called once with a "start" PullProgress that counts and sizes the source files, before they are checked and loaded. Waits for the load job and returns its summary. Without the altar[bigquery] extra, raises an ImportError that names it.

owned_dataset_storage_options

owned_dataset_storage_options(
    label: str,
    dataset: str | PathLike[str],
    *,
    storage_options: Mapping[str, Any] | None = None,
    connection: Connection | None = None,
) -> dict[str, Any] | None

Return the fsspec storage options that read an owned dataset at dataset, a path or a URI.

With a connection, they are the connection's options for dataset's URI; a connection returns none for a local path. Otherwise they are storage_options, or None when that is not given, and the provider's ambient credentials apply. label names the data in the error raised when both are given: a connection supplies the storage options, so a source takes one or the other. A connection that cannot reach dataset's scheme raises its CapabilityNotSupported.

unknown_options_message

unknown_options_message(
    label: str,
    from_options: Callable[..., object],
    unknown: Iterable[str],
    *,
    code_only: Sequence[str] = (),
) -> str

Return the error message for run spec options that from_options does not take.

The message names the unknown options, sorted, and lists the keyword-only parameters of from_options in their declared order. label names the data, such as "AlphaMissense". Parameters named in code_only take Python objects a run spec cannot spell: they are left out of the list and named in a closing clause.

A source raises ValueError with it::

if unknown:
    raise ValueError(unknown_options_message("AlphaMissense", cls.from_options, unknown))
link_gene_record(
    link: VariantGeneLink | VariantGeneEvidenceRecord,
) -> dict[str, Any]

Return one link_genes record: the LINK_GENE_FIELDS of link, with None for what it lacks.

link is a query's VariantGeneLink or the stored VariantGeneEvidenceRecord it was made from; both carry the same fields.

link_genome_build(genome_build: str) -> GenomeBuild

Return the link contract's GenomeBuild for a score-identity build name such as "hg38".

project_variant_gene_links(
    links: Iterable[Mapping[str, Any] | VariantGeneLink],
) -> dict[str, Any]

Project one variant's links into the contract's columns; an empty links gives {} (no row).

Each link is a link_genes record, a mapping with the LINK_GENE_FIELDS as link_gene_record returns it and a Parquet link dataset stores it, or a VariantGeneLink, which is converted with link_gene_record; every reader projects through this one function. Links are ordered strongest first, honoring each score's score_direction: a link whose score is missing or whose direction is unknown cannot be ranked and sorts last. Ties, and links of different directions in one relation, resolve by record_id. The link_top_* columns describe the first link when it can be ranked.

decode_link_cursor(
    query: VariantGeneLinkQuery,
) -> str | None

Decode and validate query.cursor, returning its source record id.

encode_link_cursor(
    query: VariantGeneLinkQuery, last_record_id: str
) -> str

Encode an opaque cursor bound to the query's locus and filters.

get_variant_gene_link_source_registry() -> (
    PluginRegistry[type[VariantGeneLinkSource]]
)

Return the registry for altar.variant_gene_link_sources adapters.

normalize_chromosome

normalize_chromosome(chromosome: str) -> str

Return a canonical chromosome label without a chr prefix.