Evidence sources¶
altar.sources defines annotation sources and variant–gene link sources. Evidence bindings and applications that serve
their own tables import from it. Add an annotation source shows how to write one.
Stable, except the published-dataset, dataset-builder, link-dataset and position-score names, which are provisional. Provisional names lists them.
sources
¶
Stable annotation-source and variant-to-gene evidence contracts.
This is the supported import surface for bindings backed by precomputed data, warehouses, or services. The exported records retain source evidence without forcing it into a model result.
ANNOTATION_IDENTITY_COLUMN
module-attribute
¶
The column a writable lake stamps with its source's AnnotationSourceIdentity.identity_hash.
An identity-scoped BigQueryAnnotationSource writes it on every row and deduplicates and reads only rows that
carry its own hash, so a data release bump re-annotates instead of serving cached values of the old release.
BigQueryAnnotationSource.add_annotations_from_uri stamps it on files it loads; a host that loads such a lake
with its own job stamps the same column. No annotation column may use this name.
LINK_DATASET_METADATA
module-attribute
¶
LINK_DATASET_METADATA = ParquetReleaseMetadata(
prefix="altar_links.",
distribution="altar",
label="variant-gene link",
command="a link dataset builder (write_link_dataset)",
layout="variant-gene-links/1",
)
How every file of a link dataset records its release: the fields link_dataset_release_fields names, under the
altar_links. prefix, and the layout name and version, variant-gene-links/1. Provisional.
BUILDER_VERSION_METADATA_FIELD
module-attribute
¶
The metadata field that records the installed version of the binding that built a dataset.
DEFAULT_INPUT_ATTEMPTS
module-attribute
¶
How many times a builder reads a remote input, or uploads a built file, before it gives up (BuildOutput).
DEFAULT_STDIN_NAME
module-attribute
¶
The output name standard input takes when the caller gives none.
GENOME_BUILD_METADATA_FIELD
module-attribute
¶
The metadata field that records the genome build of a dataset's coordinates.
LAYOUT_METADATA_FIELD
module-attribute
¶
The metadata field that records which of a binding's file layouts a dataset has (ParquetReleaseMetadata.layout).
RELEASE_METADATA_FIELD
module-attribute
¶
The metadata field that records the upstream release a dataset was built from.
STDIN
module-attribute
¶
The input name that reads standard input, so a host can stream a VCF from storage the builder never opens.
VARIANT_ID_COLUMN
module-attribute
¶
The column every built file keys its rows by; by default check_unique_keys rejects a value that occurs twice.
POSITION_SCORE_METADATA
module-attribute
¶
POSITION_SCORE_METADATA = ParquetReleaseMetadata(
prefix="altar_positions.",
distribution="altar",
label="position score",
command="the track's position score builder",
layout="position-scores/1",
)
How every file of a position score dataset records its release: the fields position_score_release_fields names,
under the altar_positions. prefix, and the layout name and version, position-scores/1. Provisional.
LINK_GENE_FIELDS
module-attribute
¶
LINK_GENE_FIELDS = (
AnnotationColumn(
name="gene_id", dtype="str", label="Gene ID"
),
AnnotationColumn(
name="gene_namespace",
dtype="str",
label="Gene Namespace",
),
AnnotationColumn(
name="gene_symbol", dtype="str", label="Gene Symbol"
),
AnnotationColumn(
name="score", dtype="float", label="Link Score"
),
AnnotationColumn(
name="score_direction",
dtype="str",
label="Score Direction",
),
AnnotationColumn(
name="biosample", dtype="str", label="Biosample"
),
AnnotationColumn(
name="biosample_id",
dtype="str",
label="Biosample ID",
),
AnnotationColumn(
name="regulatory_element_id",
dtype="str",
label="Regulatory Element ID",
),
AnnotationColumn(
name="element_type",
dtype="str",
label="Element Type",
),
AnnotationColumn(
name="source_id", dtype="str", label="Source ID"
),
AnnotationColumn(
name="source_name", dtype="str", label="Source Name"
),
AnnotationColumn(
name="method", dtype="str", label="Method"
),
AnnotationColumn(
name="data_release",
dtype="str",
label="Data Release",
),
AnnotationColumn(
name="record_id",
dtype="str",
label="Link Record ID",
),
AnnotationColumn(
name="source_record_id",
dtype="str",
label="Source Record ID",
),
AnnotationColumn(
name="source_dataset",
dtype="str",
label="Source Dataset",
),
)
The fields of one link_genes record: the link's gene, score, biosample and provenance.
VARIANT_GENE_LINK_ANNOTATION_CONTRACT
module-attribute
¶
VARIANT_GENE_LINK_ANNOTATION_CONTRACT = AnnotationContract(
source_id=VARIANT_GENE_LINK_ANNOTATION_SOURCE_ID,
schema_version="1.0.0",
columns=(
AnnotationColumn(
name="link_top_gene",
dtype="str",
label="Top Linked Gene",
),
AnnotationColumn(
name="link_top_gene_symbol",
dtype="str",
label="Top Linked Gene Symbol",
),
AnnotationColumn(
name="link_top_score",
dtype="float",
label="Top Link Score",
),
AnnotationColumn(
name="link_gene_count",
dtype="int",
label="Linked Gene Count",
),
AnnotationColumn(
name="link_count",
dtype="int",
label="Link Count",
),
AnnotationColumn(
name="link_genes",
dtype="struct",
label="Linked Genes",
repeated=True,
fields=LINK_GENE_FIELDS,
),
),
)
The variant-gene link annotation columns, version 1.0.0.
link_genes holds one record per link, strongest first (see project_variant_gene_links), so its first element
is the top link. link_top_gene is that link's gene ID, link_top_gene_symbol its symbol when the source
recorded one, and link_top_score its score; the three are absent when no link has a usable score.
link_gene_count counts the distinct linked gene IDs and link_count the link records. A variant with no link
has no row.
VariantIdentityError
¶
Bases: ValueError
A variant key is malformed, non-canonical, or disagrees with its locus.
VariantKey
dataclass
¶
Canonical one-based biallelic key serialized as chr:pos:ref:alt.
Fields follow the rules in this module's documentation. The class normalizes spelling but does not normalize biological representation. Callers must left-align and trim indels against the selected reference before constructing a key when equivalence across representations matters.
parse
classmethod
¶
parse(variant_id: str) -> VariantKey
Parse and normalize a four-field textual key.
require_canonical
classmethod
¶
require_canonical(variant_id: str) -> VariantKey
Parse variant_id and reject aliases instead of silently rewriting them.
from_fields
classmethod
¶
from_fields(
chromosome: str,
position: int,
reference_allele: str,
alternate_allele: str,
) -> VariantKey
Construct a canonical key from decomposed one-based locus fields.
AllelicKey
dataclass
¶
AllelicKey(
variant_id: str,
chromosome: str,
position: int,
reference_allele: str,
alternate_allele: str,
)
One backend-ready biallelic variant key.
variant_id is the caller's stable identity and is returned unchanged. The remaining fields are the
values used by the backend's physical lookup; a binding may therefore add or remove a chromosome prefix
while retaining the original variant_id. allelic_key fills them with the minimal representation of
variant_id, so two keys with different variant_id values can share one physical key.
AllelicKeyColumns
dataclass
¶
Physical column names used to join a record collection to requested variants.
as_tuple
¶
Return key columns in chromosome, position, reference, alternate order.
AllelicRecord
dataclass
¶
One source-native record matched to the caller's variant_id.
AllelicRecordBackend
¶
Bases: Protocol
Retrieve source-native records without owning their biological interpretation.
fetch
async
¶
fetch(query: AllelicRecordQuery) -> Sequence[AllelicRecord]
Return all matching records, restricted to query.keys and query.fields.
Zero, one, or many records may match each variant. Returned variant_id values must come from the
request, fields must equal the requested projection, and values must not be aggregated or reinterpreted.
Two keys with different variant_id values and one physical key each get every matching record.
AllelicRecordQuery
dataclass
¶
AllelicRecordQuery(
keys: tuple[AllelicKey, ...], fields: tuple[str, ...]
)
A projected, batch lookup against one source-native record collection.
ParquetAllelicRecordBackend
¶
ParquetAllelicRecordBackend(
dataset: str | PathLike[str],
*,
key_columns: AllelicKeyColumns,
hive_partitioning: bool = True,
storage_options: Mapping[str, Any] | None = None,
batch_size: int = DEFAULT_PARQUET_FETCH_BATCH_SIZE,
max_concurrent_files: int = DEFAULT_PARQUET_MAX_CONCURRENT_FILES,
)
Query any Parquet record collection through DuckDB, on the local disk or where it is hosted.
The adapter knows only the physical allelic key columns. Projected fields are supplied per request by the scientific binding, so one instance shape works for transcript-level AlphaMissense rows, gene-level SpliceAI rows, and future precomputed sources.
dataset is one Parquet file or a directory of them, given as a local path or as a URI: file://, or a
remote gs://, s3://, az://, http(s)://, hf:// or memory:// URI that an fsspec
filesystem reads in place. storage_options is passed to that filesystem unchanged, for non-secret options
such as {"token": "anon"} or a requester-pays project; provider credentials stay ambient. A local dataset
needs no fsspec and ignores storage_options. dataset is kept as a Path for a local dataset and as the URI
otherwise.
A lookup reads only what its keys can match. With hive_partitioning, a file under a
<chromosome column>=<value> directory is opened only when a key is on that chromosome, and is searched for
that chromosome's keys alone. Each file is scanned for its keys' chromosomes and exact positions, so Parquet
row-group statistics skip every row group that holds none of them; the builders write rows sorted by position
within a chromosome, which keeps those statistics tight, and in small row groups, so a point lookup reads one
small row group per hit. The exact allelic key join decides which rows match.
A fetch reads its keys in consecutive batches of at most batch_size, so a cohort-scale request never becomes
one query with a parameter per position. Within a batch, each file the keys can be in is read by its own query,
up to max_concurrent_files at once, so a lookup spread over many files waits for the slowest file rather than
for the sum of them. Records come back in request order across batches and files; a key matched in several
files has its records in the files' sorted order.
One DuckDB database serves every fetch of an instance, and concurrent fetches are safe. A remote dataset is treated as an unchanging release: its file list and each file's size are fetched once per instance, and each file's footer and every byte range already read are reused by later queries rather than fetched again, so the batches after the first, and later fetches, read only row groups they have not read. A local dataset is listed on every fetch, as before.
fetch
async
¶
fetch(query: AllelicRecordQuery) -> list[AllelicRecord]
Return requested native records without blocking the event loop.
close
¶
Close the DuckDB database the backend opened, if any; a second call does nothing.
A source that built this backend closes it from its own close. A later fetch opens a new database.
SqliteAllelicRecordBackend
¶
SqliteAllelicRecordBackend(
database: str | Path,
*,
table: str,
key_columns: AllelicKeyColumns,
)
Query any SQLite table containing source-native allelic records.
A fresh connection is opened for each fetch on its worker thread, which keeps the adapter safe to call
from an async application without sharing a synchronous sqlite3.Connection across threads.
fetch
async
¶
fetch(query: AllelicRecordQuery) -> list[AllelicRecord]
Return requested native records without blocking the event loop.
AnnotationDependencyError
¶
Bases: ValueError
Raised when the configured annotation sources cannot satisfy a model's annotation dependency.
AnnotationGenomeBuildError
¶
Bases: AnnotationDependencyError
Raised when annotation sources, or sources and models, declare different genome builds.
AnnotationUnavailableWarning
¶
Bases: UserWarning
Emitted when sources are not joined because they serve another assembly than the variants.
It names the dropped sources, whose columns read as missing. It is also emitted for a model's dependency that no source can supply on the variants' assembly, because the source classes implementing its contract all leave that assembly out; the model's prioritization then treats the columns as unknown. Configure a source that serves the variants' assembly to supply them.
ResolvedAnnotationDependency
dataclass
¶
ResolvedAnnotationDependency(
model_id: str,
dependency: AnnotationDependency,
source_names: tuple[str, ...],
identity: AnnotationSourceIdentity | None,
unavailable: tuple[str, ...] = (),
)
How one model's AnnotationDependency is satisfied.
source_names are the configured sources that supply the dependency's columns, in configuration order.
identity is the identity of the source that declares the dependency's contract, or None when the
columns were matched by name only (an UndeclaredAnnotationContractWarning was emitted) or are unavailable.
unavailable names the dependency columns no source can supply on the variants' assembly; they read as
missing (an AnnotationUnavailableWarning was emitted).
UndeclaredAnnotationContractWarning
¶
Bases: UserWarning
Emitted when a dependency's columns are supplied by sources that do not declare its contract.
The columns are used by name, so materialization proceeds, but nothing states that their values follow the
contract the model depends on. Declare it with identity=CONTRACT.identity(release=..., genome_build=...)
on a generic source, or by overriding AnnotationSource.identity().
AnnotationColumn
dataclass
¶
AnnotationColumn(
name: str,
dtype: AnnotationDtype,
label: str,
repeated: bool = False,
fields: tuple[AnnotationColumn, ...] = (),
)
One per-variant annotation column a source provides, such as region_type or ccre_id.
dtype, together with repeated and fields, gives the column's logical type; each backend maps that
type to its own physical form. A column with repeated=True is an array of dtype. A column with
dtype="struct" is a record whose fields are declared in fields, usually with repeated=True for an
array of records such as nearest_genes. BigQuery stores these as native types (ARRAY<STRING>,
ARRAY<STRUCT<…>>). A backend without native array or record support, such as SQLite, stores them as a
JSON blob instead. That JSON form is the backend's substitute, not the column's logical type.
AnnotationContract
dataclass
¶
AnnotationContract(
source_id: str,
schema_version: str,
columns: tuple[AnnotationColumn, ...],
)
A published, versioned column set that any source may implement.
source_id is a globally namespaced name, such as org.kundajelab.altar.annotation.regions, and is what a
model's AnnotationDependency.source_id refers to. schema_version is a semantic version: a source whose
contract has the same major version and an equal or later minor version can satisfy a dependency, because
minor versions only add columns. columns is the column set every implementation declares.
A contract does not name a data release or a genome build. Several sources can implement one contract, such
as a shipped binding and a host's own warehouse table; each states the release and build it serves through
identity(release=..., genome_build=...).
identity
¶
identity(
*, release: str, genome_build: str
) -> AnnotationSourceIdentity
Return the identity of a source that serves release of this contract on genome_build.
AnnotationIdentityError
¶
Bases: ValueError
Raised when a writable annotation lake would mix, or cannot vouch for, an annotation source identity.
Examples: a row carries another identity's hash; a SQLite lake records a different identity than the source
declares; a source that is not identity-scoped writes to a scoped lake; or a lake written before identities
were stored is opened by an identity-declaring source and has not yet been adopted with
backfill_annotation_identity().
AnnotationSource
¶
Bases: ABC
A per-variant reference annotation dataset.
Adapters register under altar.annotation_sources and are looked up by name. name is the registry key,
such as "variant_annotations", "alphamissense", or "none".
annotate returns annotations in a portable shape, AnnotationTable (variant_id -> {column: value}),
which merge_annotations and fetch_annotations consume. The BigQuery source returns a pandas frame
instead and usually fuses its read into the store's query through the SqlAnnotationSource capability,
skipping the portable merge. Core carries no pandas dependency, so the signatures at this boundary use
Tabular, a documented alias for Any, and the conformance suite checks the actual behavior.
A source does not have to be registered as an entry point. A host can subclass this class, or configure a
generic BigQueryAnnotationSource or SqliteAnnotationSource over its own tables, and pass the instance
to materialization next to shipped bindings. Altar checks, joins and prioritizes on the declared
annotation_columns() and identity(), never on the class or the registry name.
supported_genome_builds
class-attribute
¶
The assemblies an instance of this class can serve, or None when the class does not restrict them.
Each instance names the one assembly it serves in identity(). Materialization reads this declaration, with
annotation_contract(), to tell a dependency that no source can supply on the variants' assembly, such as region
annotations for mouse variants, from one the host forgot to configure (see resolve_annotation_dependencies).
The UCSC and GRC spellings of an assembly match.
option_descriptions
class-attribute
¶
What each option of from_options means, by option name, as altar sources shows it.
Provisional: the hook is expected to change.
Each key names a keyword parameter of from_options, which AnnotationSourceContract checks. A value is a
short phrase in lower case, such as "the copy the builder wrote, as a path or URI". An option left out keeps
the description altar sources gives every source, such as the one for genome_build, or none. A description
is not part of the source's identity.
from_options
classmethod
¶
Build this source from the plain values a run spec's [sources.<name>.options] mapping carries.
altar.run resolves a source by its registry name and calls this instead of the constructor, so a
source is declarable from a spec only when strings, numbers, booleans and paths are enough to build
it: the shipped Parquet-backed bindings take the path of a dataset their builder wrote and read the
release and genome build from its metadata, and RegionAnnotationSource takes its data directory.
The default raises CapabilityNotSupported: this source's constructor needs a client, a backend, or
typed column values that a spec cannot carry, so build it in code and pass it to altar.run.run as
annotation_sources.
published_dataset
classmethod
¶
published_dataset() -> PublishedDataset | None
Return where this binding's publisher hosts a ready-made copy of its latest release, or None.
Provisional, like PublishedDataset: the hook is expected to change along with it.
A binding whose data is openly licensed and expensive to build may declare the copy its publisher hosts,
so a user reads that release in place instead of building it, and generic tooling finds the copy through
the registry without importing the binding. The copy is the binding's own builder layout, and its
identity is the identity a source serving that release declares, whether it reads the published copy or
an owned copy of it. A source that reads it by default does so from from_options when no owned dataset
is named, resolving its options with PublishedDataset.select_copy; altar.testing.PublishedDatasetContract
checks the whole arrangement.
The default None means the binding publishes nothing: the user builds or brings the data.
annotation_contract
classmethod
¶
annotation_contract() -> AnnotationContract | None
Return the AnnotationContract every instance of this class implements, or None.
Provisional: the hook is expected to change.
It lets tooling describe a registered source without building it, which may need data, credentials or the
network: altar sources lists the contract's columns from it. A class that returns a contract promises
that each instance's identity() names that contract and its annotation_columns() include the
contract's columns, which altar.testing.AnnotationSourceContract checks. The default None means the
class declares no single contract, as a generic source whose columns come from its options does.
annotation_columns
abstractmethod
¶
annotation_columns() -> list[AnnotationColumn]
Return the columns this source adds to each materialized variant row.
Mirrors ModelPlugin.score_columns() on the model axis: each producer names the columns it
contributes by its role (score_columns / annotation_columns).
annotate
abstractmethod
async
¶
Fetch annotations for variant_ids and return them keyed by variant_id.
The result is a Mapping[variant_id, Mapping[column, value]], the shape merge_annotations and
fetch_annotations consume. A variant this source has no row for is left out of the result; the
later left-merge fills it with {}.
identity
¶
identity() -> AnnotationSourceIdentity | None
Return the contract, data release and genome build this source serves, or None.
A source that implements a published AnnotationContract should return
CONTRACT.identity(release=..., genome_build=...), and its annotation_columns() should include the
contract's columns. Materialization resolves each model's AnnotationDependency against these
identities (see resolve_annotation_dependencies). The default None means the source declares no
contract: its columns can still satisfy a dependency by name, with an
UndeclaredAnnotationContractWarning.
add_annotations
async
¶
add_annotations(rows: Tabular) -> InsertResult
Write annotation rows into this source.
The default raises CapabilityNotSupported: most sources are read-only reference datasets that core
never writes to. Only a writable ingest source overrides this.
get_unannotated_variants
async
¶
Return the subset of variant_ids this source has no annotation row for.
A writable source uses this before ingest so it does not re-annotate variants it already has, the
same role get_unscored_variants plays for scores. The portable contract takes variant_ids in and
returns the unannotated variant_ids. A writable source that declares an identity() counts only rows
written under that identity, so a new data release reports every variant unannotated
(altar.testing.WritableAnnotationSourceContract checks this). A source may omit ids its write path can
never store, such as a non-canonical variant_id, since reporting them would ask every job to annotate them
again.
Warehouse-scale scoping — pruning against a temp-table relation or a chr clustering key instead of
binding millions of ids as an array parameter — is a SQL concern, not part of this backend-neutral
contract. A SQL-backed source widens this signature with those parameters on its own concrete method
(BigQueryAnnotationSource.get_unannotated_variants takes variants_relation / chromosomes), the
same way ScoreStore.get_unscored_variants keeps its variants_relation off the base and only the
BigQuery store declares it. The temp-table refs and clustering keys stay behind the SQL adapter.
The default raises CapabilityNotSupported. A precomputed reference dataset such as AlphaMissense or
CADD is never written through core, so it has nothing to deduplicate; only a writable ingest source,
paired with add_annotations, overrides this.
prioritize_predicate
¶
prioritize_predicate() -> Predicate | None
Return the predicate that gates the prioritized flag, or None.
A source that prioritizes variants, such as AlphaMissense, returns a Predicate evaluated over its
own per-variant annotation columns. A source that only contributes columns without driving
prioritization, such as region_type or a gnomAD frequency, returns None (the default).
This is the same Predicate type a ModelPlugin uses. It renders to SQL for the store's materialize
step or to Python for the reference driver, so a source can gate prioritized with no model,
model_id, or scoring job. Any aggregation from per-transcript rows down to one row per variant
happens inside annotate(); the predicate reads the already-aggregated column.
close
¶
Release the connections, clients and readers this source opened itself.
The default does nothing. A source that opens a connection or creates a client, for example in
from_options, overrides it, and a second call must do nothing. A client or backend passed to the
constructor stays open: whoever created it closes it. altar.run closes the sources it builds from a run
spec when the run ends, and never one passed to it as annotation_sources.
AnnotationSourceIdentity
dataclass
¶
What a concrete source's values mean: the contract it implements plus the data release and build.
source_id and schema_version name the AnnotationContract. release is the source's data release
label, such as "4.1.1-genomes"; it must change whenever the values the source returns could change.
genome_build is the assembly of the variant IDs the source answers for, spelled as in score identity
("hg38"). Surrounding whitespace is stripped from release and genome_build, and both must be
non-empty.
identity_hash is a deterministic hash of the four fields, suitable as a cache or deduplication key.
identity_hash
property
¶
Return sha256:<hex> over the canonical JSON of the four identity fields.
from_dict
classmethod
¶
from_dict(
data: Mapping[str, Any],
) -> AnnotationSourceIdentity
Rebuild an identity from to_dict() output, rejecting missing or unknown fields.
IdentityBackfill
dataclass
¶
What a writable lake's backfill changed: rows stamped, and rows deleted.
Returned by backfill_annotation_identity on BigQueryAnnotationSource and SqliteAnnotationSource, and by
BigQueryAnnotationSource.backfill_identity_columns (the locus columns).
non_canonical counts the legacy rows a BigQuery backfill left unstamped because their variant_id is not
canonical, such as chr9:100:C:C (REF equal to ALT) or 9:100:C:T. No canonical lookup reaches them, so
they are never served or counted; find and delete them. A SQLite backfill adopts the file as a whole and
always reports 0.
UnscopedAnnotationReadWarning
¶
Bases: UserWarning
Emitted when a source that is not identity-scoped reads a table that has ANNOTATION_IDENTITY_COLUMN.
An identity-scoped writer stamps that column, so the table may hold rows of several identities, such as two
data releases. A source that is not scoped reads and counts all of them: a variant can be served an
arbitrary release's values, and deduplication counts a variant annotated under another release as
annotated. Give the source the identity it serves and identity_scoped=True; a reader with its own
relation= must also filter with the {identity_filter} token.
BigQueryAnnotationSource checks its table's schema for the column before it first reads or deduplicates
against the table. It does not check a read through relation=, whose tables only the SQL names. Writes
are not warned about: an unscoped write to a scoped lake raises AnnotationIdentityError.
BigQueryAnnotationSource
¶
BigQueryAnnotationSource(
client: Any,
table_id: str,
columns: Sequence[AnnotationColumn],
*,
name: str | None = None,
prioritize_predicate: Predicate | None = None,
relation: str | None = None,
genome_default: str | None = None,
identity: AnnotationSourceIdentity | None = None,
identity_scoped: bool = False,
)
Bases: SqlAnnotationSource
Your own per-variant values, from a BigQuery table. Build it in code with a client and the table's columns.
It reads one table (project.dataset.table) keyed on variant_id, with one column per injected
AnnotationColumn. It is a prioritizing source when a
prioritize_predicate is injected; the passive/prioritizing distinction is configuration, not a subclass.
It shares the store's SQL backend, so it implements SqlAnnotationSource: the BigQuery store fuses its
sql_join JoinSpec into the single materialize query instead of round-tripping through annotate().
A source constructed with identity_scoped=True (which needs an identity) is an identity-scoped lake.
Every row it writes carries the identity's identity_hash in the _altar_annotation_identity column
(ANNOTATION_IDENTITY_COLUMN). Deduplication counts only rows of that hash, and reads select only rows of
that hash: the bare table through a filtering subquery, and a relation= through its required
{identity_filter} token. Bumping the identity's release therefore re-annotates variants instead of
serving the old release's cached values, and a table holding several releases still yields one row per
variant. Rows written before the table carried an identity have a NULL hash and are neither served nor counted
until backfill_annotation_identity() adopts them. Without identity_scoped, identity is metadata only.
Every source object that writes or reads the lake must be scoped the same way. A source that is not scoped
looks up its table's schema, a metadata request that bills nothing, and treats a table with the identity
column as a scoped lake. add_annotations and add_annotations_frame then raise AnnotationIdentityError
instead of writing rows without a hash. The source emits UnscopedAnnotationReadWarning, once per source
object, before it first reads the table (annotate and the store's fused queries) or deduplicates against it
(get_unannotated_variants and export_unannotated). A read through relation= is not checked, because only
its SQL names the tables it reads; deduplication and writes always use the bare table, so they are checked. If
the lookup fails, the failure is logged and the operation proceeds unchecked.
client is a google.cloud.bigquery.Client (or a compatible fake). table_id is
"project.dataset.table". columns and prioritize_predicate are the wiring that makes this a passive
or prioritizing source. name is the logical source identity recorded in prioritized_source_names
(e.g. "alphamissense"), which is distinct from the class registry key ("bigquery") and defaults to
it. The client is injected so this module needs no google-cloud-bigquery at import.
relation is an optional SQL relation that overrides the bare table_id: a subquery yielding one row
per variant_id and exposing at least the declared columns. It is how a source whose backing rows are
per-transcript (AlphaMissense and REVEL collapse to the most-damaging transcript) or that needs computed
or renamed columns (gnomAD af → af_global, TO_JSON_STRING(...) blobs) fuses in-SQL, keeping the
aggregation and projection inside the source rather than the store. A {variant_filter} token in the
string expands, in both sql_join and annotate, to variant_id IN UNNEST(@<param>), so the reference
table is pruned to the job's variants before aggregating. A {identity_filter} token expands to
_altar_annotation_identity = '<identity_hash>', so a relation over a lake holding several releases
reads only this source's rows. It qualifies like {variant_filter} does (a.{identity_filter}). An
identity-scoped source's relation must contain it, and any other source's relation must not. None uses
the bare per-variant table.
genome_default is the genome build of a writable ingest lake: the genome that add_annotations and
add_annotations_frame stamp on rows that carry none (a row naming another build is rejected), and that
both get_unannotated_variants paths and export_unannotated filter on, so the dedup only counts rows
of this build and cluster-prunes the annotation table's [genome, chr, variant_id] key. No build is
assumed: those methods raise ValueError when it is None, rather than silently deduplicating an hg19
lake against hg38 rows. It is inert for a read-only reference source, which is never written or
deduplicated.
identity declares the AnnotationContract this table implements and the data release and genome build
it holds (see AnnotationSource.identity()); a host table that implements a published contract passes
CONTRACT.identity(release=..., genome_build=...). On its own it is metadata: it does not change how rows
are written, deduplicated or read. identity.genome_build is the build label that dependency resolution
compares with score runs (for example "hg38"), while genome_default is how this table spells the
build in its stored genome column (Altar's VCF ingest writes "GRCh38"). The two are independent, and
neither is derived from the other.
identity_scoped=True makes the identity decide which rows this source writes, counts and reads (see the
class docstring). It needs identity, and a relation must then contain {identity_filter}; both are
checked here and raise ValueError. Writes stamp identity.identity_hash, and a row that carries
another hash raises AnnotationIdentityError.
client
property
writable
¶
Return the BigQuery client this source runs its queries and load jobs with.
It is read-only. Assigning the client the source already holds is accepted; any other value raises
AttributeError.
table_id
property
writable
¶
Return the annotation table, as "project.dataset.table".
A source that reads through relation= still writes to and deduplicates against this table. It is
read-only. Assigning the ID the source already holds is accepted; any other value raises
AttributeError.
connect
classmethod
¶
connect(
table_id: str,
columns: Sequence[AnnotationColumn],
*,
name: str | None = None,
prioritize_predicate: Predicate | None = None,
relation: str | None = None,
genome_default: str | None = None,
credentials: Any = None,
identity: AnnotationSourceIdentity | None = None,
identity_scoped: bool = False,
) -> BigQueryAnnotationSource
Build a source against a real BigQuery client. Requires the altar[bigquery] extra.
The source owns the client it builds here, and close releases it.
close
¶
Close the BigQuery client connect built; a second call does nothing.
A client passed to the constructor is left open: whoever created it closes it.
sql_join
¶
sql_join(ctx: SqlJoinContext) -> JoinSpec
Fuse this source into the store's materialize query as a LEFT JOIN on ctx.join_key. A BigQuery
source always shares the store's backend, so it always fuses and never returns None. relation is the
backtick-quoted table id for a per-variant table (the store projects each injected column), or the
injected aggregating or computed subquery (see __init__'s relation and JoinSpec), pruned to
ctx.variants_relation (temp table) or ctx.variant_ids_param (inline param). The alias derives from
the logical source name (alphamissense, cadd, …), which the store keeps distinct across sources.
ensure_schema
async
¶
Reconcile a writable ingest table to hold the locus identity plus the injected columns.
The identity is the variant_id key, genome, and the decomposed chr/pos/ref/alt locus that
add_annotations and add_annotations_frame both write and the dedup reads filter on, clustered by
[genome, chr, variant_id]. CREATE covers a fresh table; ADD COLUMN IF NOT EXISTS reconciles a
pre-existing one by adding any missing columns (e.g. a table provisioned out of band). BigQuery cannot
add a NOT NULL column, so identity columns added to an old table are nullable. Its earlier rows have
no genome, so the dedup reports them unannotated, and because writes append, re-annotating them would
leave a second row per variant. Run backfill_identity_columns once after this migration, before any
dedup; ensure_schema never rewrites existing rows itself.
An identity-scoped source (see the class docstring) also adds a nullable _altar_annotation_identity
STRING column. Added to an existing table, it is NULL on every earlier row, so those rows are neither
served nor counted as annotated until backfill_annotation_identity adopts them. Without that
backfill, the next deduplication re-annotates the whole lake. The clustering is unchanged.
backfill_identity_columns
async
¶
backfill_identity_columns() -> IdentityBackfill
Stamp the locus identity on rows written before the table carried it, as one transaction.
This is the explicit, one-time migration for a table that predates the identity columns (see
ensure_schema, which must run first to add them; it never runs this). It assumes every legacy row, one
with a NULL genome, belongs to genome_default's build, which is therefore required. Only a legacy row
whose variant_id is canonical (_CANONICAL_VARIANT_ID_PATTERN, the RE2 form of
VariantKey.require_canonical) is touched:
- If its variant already has a row of
genome_default's build (it was re-annotated before this ran), the legacy row is deleted. - Otherwise exactly one legacy row per
variant_idis kept, stamped withgenome_defaultand thechr/pos/ref/altparsed from thevariant_id, and the others are deleted. The survivor is the most recent bycreated_atwhen the table has that column, then the first by the row's JSON text, so a rerun on the same rows keeps the same one.
A legacy row with a non-canonical variant_id (1:9:A:T, chr6:-3:A:T, chr9:5:C:C) keeps its NULL
genome: no canonical lookup can reach it, and stamping it would invent a locus. The work runs as one
BigQuery multi-statement transaction (temp-table snapshot, DELETE, INSERT), so a concurrent write
cannot interleave with it; stop ingest into the table while it runs and resume afterwards. A second run
finds no canonical legacy rows and changes nothing. It returns how many rows were stamped and deleted, and
as non_canonical how many legacy rows it left alone for their variant_id.
backfill_annotation_identity
async
¶
backfill_annotation_identity() -> IdentityBackfill
Adopt the rows written before this lake stored an annotation identity, as one transaction.
This is an explicit, reviewed one-time migration, not something Altar runs on its own. It asserts that
every legacy row of genome_default's build, one whose _altar_annotation_identity is NULL, was produced
by this source's configured identity (the same contract, data release and genome build). Run it only
with the identity of the release that actually built those rows. Adopting them under a newer release
would serve old values under the new identity, which is the error identity scoping exists to prevent.
The source must be identity-scoped (identity_scoped=True) and have a genome_default, and
ensure_schema must have added the column. Rows with a NULL genome are not touched: run
backfill_identity_columns first for those. Of the legacy rows of this build:
- If its variant already has a row stamped with this identity (it was re-annotated before this ran), the legacy row is deleted.
- Otherwise exactly one legacy row per
variant_idis kept and stamped with the identity hash, and the others are deleted. The survivor is chosen as inbackfill_identity_columns: the most recent bycreated_atwhen the table has that column, then the first by the row's JSON text.
A legacy row whose variant_id is not canonical (_canonical_variant_id_sql), such as chr9:5:C:C
(REF equal to ALT) or 9:5:C:T, is neither stamped nor deleted. It stays unstamped, so the identity never
serves or counts it, and the result reports how many there are as non_canonical so you can delete them.
Once this has run, _altar_annotation_identity IS NULL AND genome = <genome_default> selects exactly
those rows.
Legacy rows of another genome and rows stamped with any identity are left alone. Like
backfill_identity_columns it runs as one BigQuery multi-statement transaction (temp-table snapshot,
DELETE, INSERT); stop ingest into the table while it runs. A second run finds no canonical legacy rows
of this build and changes nothing. It returns how many rows were stamped and deleted, and how many it left
unstamped as non_canonical.
Cost: the snapshot reads every column of the legacy rows, and on-demand pricing bills the DELETE for
every column of the whole table, so when most rows are legacy the transaction bills about three times the
table's logical size and rewrites all of its storage. BigQuery re-clusters the rewritten rows in the
background at no charge. The replaced storage stays in time travel and fail-safe for their configured
windows, which a dataset on physical storage billing pays for.
add_annotations
async
¶
add_annotations(rows: Tabular) -> InsertResult
Append per-variant annotation rows, each a {"variant_id": ..., <column>: value, ...} mapping.
The write is append-only (WRITE_APPEND), so deduping is the caller's job (pre-flight via
get_unannotated_variants). Like add_annotations_frame, every row is written with its genome and
decomposed locus, so both write paths fill the identity columns the dedup reads filter on. variant_id
must be canonical: a non-canonical one raises VariantIdentityError rather than being written where no
read can find it. chr/pos/ref/alt are derived from it, and any the row carries must name the same
variant. genome is the row's own, else genome_default (see __init__). An identity-scoped lake also
stamps _altar_annotation_identity with its identity hash; a row that carries another hash raises
AnnotationIdentityError. The table must already have that column (ensure_schema). A source that is not
identity-scoped raises AnnotationIdentityError when its table has that column (see the class docstring).
add_annotations_frame
¶
add_annotations_frame(frame: Any) -> InsertResult
Ingest a per-variant annotation frame columnar, the annotation counterpart of add_scores_frame.
It re-derives the variant_id (chr:pos:ref:alt) join key from the locus, overwriting any
present-but-null passthrough id and rejecting a non-canonical locus with VariantIdentityError, stamps
genome_default on a frame without a genome column (and rejects a genome column naming another build
when genome_default is set), stamps an identity-scoped lake's identity hash (rejecting a frame whose
_altar_annotation_identity names another), stamps created_at, and casts each declared scalar
column to its physical dtype (nullable Int64, bool, or float64). repeated, struct, str, and
json columns pass through untouched: BigQuery's load_table_from_dataframe (Parquet) maps a
list-of-scalar to ARRAY<…> and a list-of-dict to ARRAY<STRUCT<…>>. It drops any af_* columns the
caller's frame carries, since allele-frequency columns live in a separate source joined at materialize,
so ALLOW_FIELD_ADDITION can't resurrect them. The write is append-only (WRITE_APPEND), so deduping is
the caller's job (pre-flight via get_unannotated_variants), as in the score store. A source that is not
identity-scoped raises AnnotationIdentityError when its table has _altar_annotation_identity (see the
class docstring).
add_annotations_from_uri
async
¶
add_annotations_from_uri(
uri: str | Sequence[str],
*,
expected_identity: AnnotationSourceIdentity
| None = None,
verify_identity: bool = True,
staging_dataset: str | None = None,
expiration: timedelta = _URI_LOAD_EXPIRATION,
) -> InsertResult
Load newline-delimited JSON annotation rows from uri into this lake, skipping variants it already has.
This is the warehouse-scale write for files an annotation job wrote to object storage, such as the region
annotate command's JSONL. uri is any source URI, or list of URIs, a BigQuery load job accepts, such as
gs://bucket/run-7/*.jsonl. Each row needs a canonical variant_id and may carry any of the declared
columns. Fields that are not declared columns, such as the annotate JSONL's variant_type and af_*
fields, are ignored.
expected_identity is the identity the rows were produced under, typically read from the job's identity
sidecar with load_annotation_identity. When it is given, the load is refused with
AnnotationIdentityError before anything is loaded if it differs from this source's identity, or if the
source declares none. An identity-scoped source requires it, because it stamps its own identity on every
row: without a check, files an outdated annotation image produced would be stored as the new release.
verify_identity=False waives that requirement for a caller that has checked the files' identity itself.
The rows are loaded into a scratch table in staging_dataset (a project.dataset; the lake's own dataset
by default) with the declared columns' types, so repeated and struct columns load natively. The scratch
table expires after expiration as a backstop and is dropped when the call returns or fails. Then:
- Every
variant_idmust be canonical (chr1:100:A:T, with different alleles). The annotate command copies the id from its input's fifth column and writesnullwithout one;export_unannotatedwrites that column. chr,pos,refandaltfields the rows carry must name the same variant as thevariant_id, as inadd_annotations:posexactly, the alleles ignoring case, andchrwith or without itschrprefix (andMTforM). Rows without them are not checked.- On an identity-scoped lake, a row's own
_altar_annotation_identityfield must be empty or this source's identity hash.
A row that fails a check raises VariantIdentityError (or AnnotationIdentityError for the identity field)
and nothing is inserted. Otherwise one INSERT … SELECT appends the rows, in the stored form of
add_annotations:
genomeisgenome_default, which is required, andchr/pos/ref/altare parsed from thevariant_id.- An identity-scoped lake stamps
_altar_annotation_identitywith its identity hash. - A variant that already has a row of this build (and, when identity-scoped, of this identity) is
skipped, so loading the same files twice adds nothing. The match is on
variant_id,genomeand the identity hash only, never on the storedchr, so a row written with anotherchrspelling still counts. That anti-join reads those three columns of every lake row of the build: on an 80-million-row lake, about 8.6 GB per load. - Rows of the same
variant_idwithin the load collapse to one, the first by the row's JSON text.
It returns InsertResult(added=<rows inserted>, skipped=<loaded rows not inserted>); skipped counts both
variants the lake already had and duplicate rows within the load. The lake must already have the columns
(ensure_schema). Concurrent loads of the same variants into one lake can each insert a row, as with the
other write paths; serialize them. A source that is not identity-scoped raises AnnotationIdentityError
before loading anything when its table has _altar_annotation_identity (see the class docstring).
get_unannotated_variants
async
¶
get_unannotated_variants(
variant_ids: Sequence[str],
*,
variants_relation: str | None = None,
chromosomes: Sequence[str] | None = None,
) -> list[str]
Return the subset of variant_ids this lake has no annotation row for. This is the ingest dedup that
lets an annotation job annotate only cache misses, the annotation counterpart of get_unscored_variants,
differing only in the missing model_id (annotations are not per-model). It binds @variant_ids inline
by default. Pass variants_relation (a backtick-quoted temp-table ref carrying variant_id) to prune at
warehouse scale as materialize does, for a variant set too large to bind as an array param. On the
relation path the full input lives only in the temp table, so the unannotated rows are read straight off
a LEFT JOIN … IS NULL dedup and variant_ids may be empty. chromosomes then prunes the annotation
table's chr cluster key; it is ignored on the inline path, whose small array binds directly. Both paths
count only rows of genome_default's build, so it is required, and an identity-scoped lake counts only
rows stamped with its identity hash, so a release bump reports every variant unannotated. It returns
variant_ids, and the caller decomposes them back to loci (chr:pos:ref:alt) for the annotation-job
input file.
Neither path ever returns a variant_id that is not canonical (VariantKey.require_canonical), such as
chr9:100:C:C (REF equal to ALT), 9:100:C:T, or a NULL id in the relation. No write path stores a row
for one, so reporting it would ask every job to annotate it again. export_unannotated leaves out the
same ids and logs how many it left out.
export_unannotated
async
¶
export_unannotated(
*,
destination_uri: str,
variants_relation: str,
chromosomes: Sequence[str] | None = None,
) -> int
Export a job relation's unannotated variants to a headerless object-store TSV; return the count.
Each line is chr, pos, ref, alt, variant_id, the input the annotate command
(altar.variants.annotation.annotate) reads: it copies the fifth column into each JSONL row's
variant_id, which add_annotations_from_uri requires. chr, pos, ref and alt are split from the
canonical variant_id, not read from the relation's own locus columns, so every line names the variant
its id names and the loader's locus check cannot reject it. The relation needs only variant_id.
destination_uri needs one * wildcard, and BigQuery chooses how many files it writes; any of them may be
empty. The annotate command reads one file, so annotate each shard (an empty one yields an empty JSONL) or
concatenate them first, then load all the JSONL with one add_annotations_from_uri("gs://…/*.jsonl",
expected_identity=…). Give every job a fresh prefix: overwrite=true replaces only the files this export
writes, never stale files an earlier export left under the same wildcard, and an export of 0 variants
writes nothing at all.
It counts and exports the same cache misses as get_unannotated_variants with variants_relation: only
rows of genome_default's build, and, for an identity-scoped lake, of its identity hash, are annotated.
Like it, it leaves out every unannotated variant whose variant_id is not canonical
(VariantKey.require_canonical), such as chr9:100:C:C (REF equal to ALT), 9:100:C:T, or a NULL id,
because the loader refuses a whole file for one such row. The return value counts only exported rows.
When it left any out, it logs a warning with their count and an example, so you can delete them from the
relation's source.
annotate
async
¶
Return {variant_id: {column: value}} for the requested variants this table has rows for. A variant
with no row is absent, and the left-merge then yields {} for it (see merge_annotations). A declared
scalar json column that the relation returns as JSON text, such as TO_JSON_STRING(...), is decoded to
its JSON value, as SqliteAnnotationSource decodes its stored text. An identity-scoped lake reads only
rows of its identity, through the same relation sql_join uses.
BigQueryAllelicRecordBackend
¶
BigQueryAllelicRecordBackend(
client: Any,
table_id: str,
*,
key_columns: AllelicKeyColumns,
filters: Mapping[str, str | int | bool] | None = None,
batch_size: int = DEFAULT_FETCH_BATCH_SIZE,
)
Read source-native allelic records from one BigQuery table.
table_id is a project.dataset.table reference. key_columns names the table's chromosome, position,
reference-allele, and alternate-allele columns. filters limits every read to rows whose columns equal
fixed values. Use it when one table holds several genome builds or releases, for example
{"genome": "hg38"}. Filter values are bound as query parameters.
Each requested key must describe the locus its variant_id names. A binding may spell the chromosome
differently from the variant ID, for example 1 for chr1, but the position and alleles must match the ID's,
as written or at their minimal representation.
fetch raises ValueError for a key that does not. covered_variant_ids relies on the same rule to find
candidate variants in SQL without calling the binding.
A fetch query reads the key and requested columns of the whole table, because BigQuery can skip storage
only by clustering or partitioning. fetch therefore sends keys in batches of at most batch_size, one query
per batch. Variants returned by the most recent covered_variant_ids call are answered from memory instead.
fetch
async
¶
fetch(query: AllelicRecordQuery) -> list[AllelicRecord]
Return every record matching query.keys, with only query.fields, in request order.
covered_variant_ids
async
¶
Return the variants in variants_relation that have at least one record in this table.
variants_relation is a SQL relation with a variant_id column in chr:pos:ref:alt form, such as a
backtick-quoted table id. The match compares positions exactly and alleles ignoring case and surrounding
spaces. It compares chromosomes after removing a chr prefix and leading zeros and treating MT as M.
That comparison is looser than fetch, so it can include a variant fetch finds nothing for. It reads
each ID as written: an ID that is not in minimal representation, which fetch would trim and find, is not
covered.
The query also reads every column of the matched records and keeps them in memory. The next fetch for
a covered variant is answered from those records, with the same exact key match a query would apply, and
then releases them. If a covered variant's records are spelled differently from the requested key, for
example 1 in the table and chr1 in the key, that fetch raises ValueError rather than return no
records. A later call to this method replaces records that were never fetched. Point the backend at a
table or view without unused wide columns to keep this read small.
LinkDatasetSummary
dataclass
¶
What write_link_dataset wrote: the rows, the exact duplicates it dropped, and the files. Provisional.
BuildInput
dataclass
¶
BuildInput(
path: Path | None,
name: str,
uri: str | None = None,
storage_options: Mapping[str, Any] = dict(),
)
One input VCF: a local file, a URI read through fsspec, or standard input when both path and uri are unset.
storage_options open a URI's filesystem; they never enter equality or the text of the input.
name
instance-attribute
¶
The file name, or the name standard input stands for, which names the output and is checked by pattern.
stem
property
¶
Return the output name: name without its .vcf or .tsv, .gz, and .bgz suffixes.
label
property
¶
Return how messages name this input; a URI is named without its query, which may hold a signature.
BuildOutput
¶
BuildOutput(
output: str | PathLike[str],
*,
partition_column: str,
label: str,
metadata: Mapping[str, str],
connection: Connection | None = None,
unique_key: Sequence[str] = (VARIANT_ID_COLUMN,),
)
Where a builder writes its dataset: a local directory, or a URI that each input's files are uploaded to.
output is a local path, a file:// URI, or a URI such as gs://bucket/gnomad-4.1.1-genomes, opened with
connection.storage_options(output) or with the provider's ambient credentials when there is no connection.
Every file the build writes records metadata, its partition directories are <partition_column>=<value>,
and label names the build in messages. unique_key names the columns no two rows of a partition may share
(check_unique_keys): variant_id alone for one row per allele, or variant_id and a column such as a
transcript for a dataset with several records per allele. Provisional: the shape is expected to
change.
Use open around the build. Each input is built by build_input, which reads a remote input again from the
start when its stream fails or is cut short, at most attempts times. A local output is written in place, as
build_output writes it. For a URI output, each input is built in a local scratch directory (under
TMPDIR), its keys are checked there (check_unique_keys), and each finished file is uploaded under a
temporary name and committed, then deleted locally, so the scratch holds one input's files at a time. A
reader of the output never sees a partial file.
A URI output records its progress under _altar_build/ until the build finishes, so running the same
command again after a crash resumes it: the inputs whose files were all committed are skipped (resume), and
the others are built again. An instance is picklable, so worker processes build and upload their own inputs.
resumed
instance-attribute
¶
The names of the inputs resume skipped, because an unfinished build had already committed them.
remote
property
¶
Return whether the output is a URI that files are uploaded to, rather than a local directory.
open
¶
Prepare the output for the body of the with, which builds every input and checks the build.
A local output behaves as build_output: it must be empty or absent, or hold a previous build that
overwrite deletes first, and a failed body deletes everything the build wrote.
A URI output must be empty or absent, or hold an unfinished build of the same release (one whose
_altar_build/build.json records the same metadata, apart from the builder version), which is resumed.
With overwrite, a previous build there, finished or not, is deleted first. Either way, anything a build
does not write (files outside <partition_column>=*/*.parquet and _altar_build/) is refused, and so
is, without overwrite, a finished build or an unfinished build of another release. A failed body keeps
the files already committed, so the same command resumes the build; when the body succeeds,
_altar_build/ is deleted and the output is a finished build.
resume
¶
resume(inputs: Sequence[BuildInput]) -> list[BuildInput]
Return the inputs still to build, in order; set resumed to the names of those skipped.
An input is skipped when the unfinished build open resumes committed every file it wrote. Each one
skipped is logged. A local output, or a fresh one, skips nothing.
build_input
¶
build_input(
source: BuildInput,
attempt: Callable[[Path], Built],
*,
attempts: int = DEFAULT_INPUT_ATTEMPTS,
) -> Built
Build one input with attempt, which writes its files under the directory it is given and lists them.
attempt must delete what it wrote when it fails. When it fails on a remote input with a cut-short or
failed stream (TruncatedInputError, a stream that ended inside a gzip member, or a transient network
error), it is called again on an empty directory after a pause, up to attempts calls in all; after
that an OSError names the input and the last error. Any other error, and any error on a local file or
standard input, is raised at once.
For a local output, attempt writes into the output and its result is returned as it is. For a URI
output, it writes into a fresh scratch directory; the keys in each of its partition files are checked to be
unique, each file is uploaded and committed (each upload tried up to attempts times) and deleted
locally, the input is recorded as committed, and the result's files are replaced by the URIs of the
uploaded files. Must be called inside open, in this process or in a worker process given this instance.
check_unique_keys
¶
Reject a unique_key written twice in a partition of the output, across inputs or by trimming.
A local output is checked in full (check_unique_keys). A URI output's files were each checked before
they were uploaded, so only partitions that hold several files, from several inputs, are checked again,
by reading their key columns from the output; a build of one input per chromosome reads nothing. A
failure leaves the build unfinished: fix the inputs and rebuild with overwrite.
OpenInput
¶
OpenInput(source: BuildInput)
A text view of one input, whose compression is read from its first bytes rather than its name.
Use it as a context manager and read either text, line by line, or header_lines and then tsv_batches,
which parse the rest of the input in bulk; the two views share one stream, so a builder uses one or the other.
Standard input is left open on exit. A URI is streamed through its fsspec filesystem in blocks, never
downloaded whole. After reading to the end, check_complete rejects a bgzipped input that was cut short.
header_lines
¶
Yield the input's lines one at a time, read without reading ahead, so tsv_batches starts after them.
Stop iterating at the last header line; the lines the caller took are not parsed again. Provisional, like
BuildOutput.
tsv_batches
¶
tsv_batches(
column_names: Sequence[str],
*,
block_size: int = _BATCH_BYTES,
) -> Iterator[RecordBatch]
Parse the rest of the input as tab-separated text into record batches of string columns.
Each batch holds the lines of about block_size bytes, parsed by PyArrow's CSV reader on its own threads,
so memory stays bounded however large the input. Every line must have exactly the named columns; there is
no quoting, an empty line is skipped, and an empty field is the empty string, never null. A line with
another number of columns raises ValueError naming the input and the line. A stream that fails or ends
early raises the error the stream raised, such as the EOFError of a cut-short gzip member, which
BuildOutput.build_input retries for a remote input. Provisional, like BuildOutput.
convert_batches
¶
convert_batches(
column_names: Sequence[str],
convert: Callable[[RecordBatch, int], T],
*,
workers: int | None = None,
block_size: int = _BATCH_BYTES,
) -> Iterator[T]
Parse the rest of the input with tsv_batches and yield convert(batch, first_line) for each, in order.
first_line numbers the batch's first row among the data rows, from 1, so convert can name a line in
an error. The conversions run on workers threads (by default up to 8), which overlap because Arrow's
compute releases the GIL, while this generator reads the next batches; at most twice workers batches
are in flight, so memory stays bounded. convert must not change shared state: do what depends on the
order of the input, such as RecordOrder and PartitionedWriter, with the results as they are yielded.
An error raised by convert is raised when its batch's turn comes, and the batches after it are not
converted. Provisional, like BuildOutput.
check_complete
¶
Raise TruncatedInputError if a bgzipped input ended without bgzip's end-of-file block.
ParquetReleaseMetadata
dataclass
¶
ParquetReleaseMetadata(
prefix: str,
distribution: str,
label: str,
command: str,
legacy_prefixes: tuple[str, ...] = (),
layout: str | None = None,
)
How one binding records its data release in the key-value metadata of every Parquet file it builds.
prefix namespaces the keys, as in altar_popfreq.; the fields under it are the binding's own, such as a
version and a dataset, or RELEASE_METADATA_FIELD and GENOME_BUILD_METADATA_FIELD. write adds
BUILDER_VERSION_METADATA_FIELD, the installed version of distribution. label names the data in messages
and command is the builder a rejected file should be rebuilt with. legacy_prefixes are prefixes earlier
builders wrote: read takes a field under the current prefix first, then under each legacy prefix in turn.
layout names the file layout the binding's source reads, for a binding whose builder changed its columns or
partitions after datasets were built with the old ones. write records it as LAYOUT_METADATA_FIELD, and
read_dataset and check reject a file that records this binding's metadata but another layout, or none,
telling the reader to rebuild it, before they compare any release field: a file of the old layout would
otherwise be read wrong. A file with none of the binding's metadata is refused as before. A binding
whose files never changed layout leaves it None, and nothing is written or checked.
write
¶
Return the metadata to write into a file: fields under prefix, layout, and the builder's version.
read
¶
read(
path: str | PathLike[str],
*,
storage_options: Mapping[str, Any] | None = None,
) -> dict[str, str]
Return the fields a Parquet file records, without their prefix; the current prefix wins over a legacy one.
path is a local path or a URI, read as ParquetAllelicRecordBackend reads its dataset, with
storage_options passed to a remote filesystem. Only the file's footer is read.
read_dataset
¶
read_dataset(
dataset: str | PathLike[str],
*,
required: Sequence[str],
storage_options: Mapping[str, Any] | None = None,
) -> dict[str, str]
Return the fields the first Parquet file under dataset records, which must include required.
dataset is a file or a directory searched recursively, as a local path or a URI with storage_options
for its filesystem (see read); the first file in sorted order is read, and a binding's source then holds
the dataset to the release it names with check. For a remote dataset, the next check of the same
dataset with the same options reuses this read instead of listing the dataset and reading the footer
again, so a source that takes its release from the dataset and then checks it reads one footer. Raises
FileNotFoundError when there is no Parquet file and ValueError, naming the missing fields and the
builder to rebuild with, when the file does not record every required field or has another layout.
check
¶
check(
dataset: str | PathLike[str],
expected: Mapping[str, str],
*,
storage_options: Mapping[str, Any] | None = None,
all_files: bool = False,
) -> None
Raise ValueError if the first Parquet file under dataset, or with all_files any, is not expected.
A file fails when it records none of expected's fields, or other values for them, or another layout
than this metadata's. dataset is a file or
a directory searched recursively, as a local path or a URI with storage_options for its filesystem (see
read); a dataset with no Parquet file is left to the backend. Only footers are read.
By default only the first file in sorted order is checked, which is what a source does when it opens a
dataset: a release is identified at the release level, so a later file built from another release is
not caught there, and opening a remote dataset of many files costs one footer read rather than one per
file. When the last read_dataset of the same remote dataset has not been checked yet, its read is
reused and nothing is read. all_files reads every file's footer, as altar pull does to verify a copy
it makes.
PartitionedWriter
¶
PartitionedWriter(
output: Path,
stem: str,
*,
schema: Schema,
partition_column: str,
row_group_rows: int,
metadata: Mapping[str, str],
parquet_options: Mapping[str, Any] | None = None,
)
Write one input's rows to <output>/<partition_column>=<chromosome>/<stem>.parquet, one file per chromosome.
Each row holds every column of schema; the file records metadata in its schema and is written in row
groups of row_group_rows with zstd compression. A file is created when its chromosome's first row arrives
and is written under a .partial name until commit renames it, so a reader never sees a file in progress.
parquet_options are further keyword arguments to pyarrow.parquet.ParquetWriter, such as
use_dictionary=["chr", "score"] and column_encoding={"pos": "DELTA_BINARY_PACKED"} for a column of
increasing positions; they may override compression. Without them a file is written exactly as before the
option existed. Provisional, like append_batch.
append
¶
Buffer one row of chromosome's file.
append_batch
¶
Buffer a table or record batch of chromosome's rows, as append would buffer each row in turn.
The batch holds every column of schema, in any order and with any other columns ignored, each of a type
that casts to the schema's; a non-nullable column must hold no null. Rows keep their order, and the file's
row groups are the same as if the rows had come one at a time, so a builder may mix the two. An empty
batch creates no file. Provisional, like BuildOutput.
commit
¶
Finish every file and return their paths, in the order their chromosomes first appeared.
RecordOrder
¶
Require sorted input: each chromosome contiguous, positions non-decreasing within it, no key twice at one.
Call advance with each record's chromosome and position, then claim with each allele key it yields. A
key is (position, reference, alternate) after trimming, so a trimmed key may move past the record's
position; a key repeated across positions is caught by check_unique_keys over the finished output. kind
names the input format in messages, such as "TSV".
advance
¶
Move to the next record, rejecting one out of order.
advance_batch
¶
Move past a run of records on one chromosome at once, as advance would for each position in turn.
Rejects the same out-of-order record, with the same message, as advance called once per position.
Provisional, like BuildOutput.
claim
¶
Record an allele key at the current position, rejecting one already written there.
TruncatedInputError
¶
Bases: ValueError
A bgzipped input ended without bgzip's end-of-file block: a download or stream was cut short.
A remote input that ends this way is read again (BuildOutput.build_input); a local file is rejected.
PositionScoreAnnotationSource
¶
Bases: AnnotationSource
Annotate variants with a base-wise score track, such as phyloP, from a position score dataset. Provisional.
The source implements contract, which declares exactly one float column, and reads a dataset in the
POSITION_SCORE_METADATA layout built for that contract (its footer's track). Build it with from_dataset.
indels chooses which variants are scored. "skip" (the default) scores SNVs only, by the score at their
base. "max" scores every variant by the greatest score over the bases variant_positions gives it: an
SNV's base, a deletion's deleted bases, the two bases an insertion falls between, or another change's REF span;
unscored bases are ignored. SNVs score the same in both modes. Because "max" scores variants "skip"
leaves out, its identity's release is the dataset's release plus +indels-max.
A variant is read at its minimal representation (minimal_variant_key), and its row is keyed by the ID as
passed. A variant with no scored base, or on a contig the dataset has no partition for (an alt contig, say), has
no row. annotate reads all of a call's variants together, each needed row group once. prioritize_predicate,
evaluated over the contract's column, makes the source gate prioritized; without one it only annotates.
name is the logical source name, by default the contract's column name. A dataset is on hg38 or hg19
(supported_genome_builds), and the source's identity names the one its footer records.
release
property
¶
The identity's release: the dataset's release, plus +indels-max when indels are scored.
from_dataset
classmethod
¶
from_dataset(
dataset: str | PathLike[str],
*,
contract: AnnotationContract,
indels: str = "skip",
genome_build: str | None = None,
storage_options: Mapping[str, Any] | None = None,
prioritize_predicate: Predicate | None = None,
name: str | None = None,
) -> Self
Read the position score dataset at dataset, a local path or a URI such as gs://... or hf://....
The dataset is read in place, with storage_options passed to its fsspec filesystem. Its release and genome
build come from the first file's footer (POSITION_SCORE_METADATA). ValueError is raised for a contract
that is not exactly one float column, an indels other than "skip" or "max", a footer of another
layout or none, a footer whose track is not contract, and a genome_build other than the footer's.
OwnedDatasetCopy
dataclass
¶
An owned Parquet copy a run spec names with dataset (PublishedDataset.select_copy).
Provisional, like PublishedDataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
str | PathLike[str]
|
The copy's path or URI, as the spec gave it. |
required |
storage_options
|
Mapping[str, Any] | None
|
The fsspec options that read it, from the spec or its connection; |
required |
OwnedTableCopy
dataclass
¶
OwnedTableCopy(
table: str, release: str, connection: Connection
)
An owned BigQuery table a run spec names with table (PublishedDataset.select_copy).
Provisional, like PublishedDataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
table
|
str
|
The table, |
required |
release
|
str
|
The release label the table is read as: the spec's |
required |
connection
|
Connection
|
The connection whose |
required |
PublishedCopy
dataclass
¶
PublishedCopy(
location: PublishedLocation,
storage_options: dict[str, Any],
)
The published copy a run spec reads by default: one of a PublishedDataset's locations (select_copy).
Provisional, like PublishedDataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
PublishedLocation
|
The location to read, the primary unless the options named another. |
required |
storage_options
|
dict[str, Any]
|
The fsspec options that read it: the spec's own, a connection's, or anonymous ones. |
required |
PublishedDataset
dataclass
¶
PublishedDataset(
locations: tuple[PublishedLocation, ...],
identity: AnnotationSourceIdentity,
release_metadata: ParquetReleaseMetadata | None = None,
release_fields: Mapping[str, str] = dict(),
cluster_by: tuple[str, ...] = (VARIANT_ID_COLUMN,),
)
One release of a binding's data, published as the binding's Parquet layout at one or more public locations.
Provisional: see the module docstring.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
locations
|
tuple[PublishedLocation, ...]
|
Where the release's files are hosted, in order of preference: the first is the primary, which
a reader uses unless it names another, and the rest are mirrors holding the same files, such as a
Hugging Face copy of a Cloud Storage dataset. Names are unique. A location never enters |
required |
identity
|
AnnotationSourceIdentity
|
The contract, data release, and genome build of the published data: the identity a source serving it declares, whichever location or copy it is read from. |
required |
release_metadata
|
ParquetReleaseMetadata | None
|
How the binding's builder records the release in each file's footer, the
|
None
|
release_fields
|
Mapping[str, str]
|
The footer fields, without their prefix, that every file of this release records, such as
|
dict()
|
cluster_by
|
tuple[str, ...]
|
The columns a BigQuery table loaded from the dataset is clustered on, in order, by
|
(VARIANT_ID_COLUMN,)
|
uri
property
¶
Return the primary location's URI, the one a reader uses unless it names another location.
release
property
¶
Return the data release label the dataset serves, such as "4.1.1-genomes".
genome_build
property
¶
Return the assembly of the dataset's variant coordinates, such as "hg38".
storage_options
property
¶
Return the fsspec options that read the primary location anonymously (PublishedLocation).
location
¶
location(name: str | None = None) -> PublishedLocation
Return the location named name, or the primary (the first) when name is None.
Raises ValueError for a name the dataset does not list, naming the ones it does.
read_storage_options
¶
read_storage_options(
connection: Connection | None = None,
*,
location: str | None = None,
) -> dict[str, Any]
Return the fsspec options to read the location named location, the primary by default.
They are connection's for that location's URI, or anonymous ones; see
PublishedLocation.read_storage_options. An unknown location raises ValueError.
check_release
¶
check_release(
dataset: str | PathLike[str],
*,
storage_options: Mapping[str, Any] | None = None,
all_files: bool = True,
) -> None
Raise ValueError unless every Parquet file under dataset records this release and genome build.
dataset is a copy of this dataset, or one of its locations itself, as a local path or a URI read with
storage_options; only each file's footer is read, with release_metadata against release_fields.
Every file is checked, as verifying a copy needs; all_files=False checks only the first file in sorted
order, as a source does when it opens a dataset (ParquetReleaseMetadata.check). A dataset that declares
no release_metadata raises ValueError, since its copies cannot be checked.
select_copy
¶
select_copy(
label: str,
*,
dataset: str | PathLike[str] | None = None,
table: str | None = None,
release: str | None = None,
storage_options: Mapping[str, Any] | None = None,
connection: Connection | None = None,
location: str | None = None,
) -> PublishedCopy | OwnedDatasetCopy | OwnedTableCopy
Return the copy of this dataset's data that a source's run spec options name, and how to reach it.
A binding that publishes its data calls this from AnnotationSource.from_options with the options it was
given, so every such binding reads the same three modes by the same rules:
- With neither
datasetnortable, aPublishedCopy: the location namedlocation(the primary by default), read withstorage_optionswhen given, or else withconnection's storage options for its URI, or else anonymously (PublishedLocation.read_storage_options). - With
dataset, anOwnedDatasetCopy: an owned Parquet copy at that path or URI, read withconnection's storage options for it, or withstorage_options; with neither, the storage options areNoneand the provider's ambient credentials apply. - With
table, anOwnedTableCopy: an owned BigQuery table, read throughconnection, which it needs. A table holds no Parquet metadata, soreleasenames the release it holds, and defaults to this dataset's release, which a table loaded from it holds. The binding parses and checks the label.
Options that do not name one readable copy raise ValueError, with messages that begin with label, the
data's name, such as "gnomAD": both dataset and table; location with either, since it picks where
the published copy is read from; storage_options with a connection (which supplies them) or a table
(which needs none); release without a table, since Parquet files record their own release; a table
without a connection; and an unknown location. Nothing is read: the binding opens the copy, checks its
release, and builds its source. No option chosen here enters the source's identity.
PublishedLocation
dataclass
¶
One place a published dataset's files are hosted, under a short name a user picks it by.
Provisional: see the module docstring.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
A short lowercase name, such as |
required |
uri
|
str
|
The dataset's directory there, as a URI that an fsspec filesystem reads in place, such as
|
required |
storage_options
property
¶
Return the fsspec options that read the location anonymously: {"token": "anon"} for gs://.
read_storage_options
¶
read_storage_options(
connection: Connection | None = None,
) -> dict[str, Any]
Return the fsspec options to read the location: connection's for uri, or anonymous ones.
A connection reads the same data with other credentials, such as a project billed for requester-pays
reads; it never changes which data is read. A connection that cannot reach the URI's scheme raises
CapabilityNotSupported.
BigQueryLoadSummary
dataclass
¶
BigQueryLoadSummary(
table: str,
source_uri: str,
job_id: str | None,
rows: int | None,
write_disposition: str,
clustering_fields: tuple[str, ...],
identity: AnnotationSourceIdentity,
labels: Mapping[str, str] = dict(),
)
What pull_to_bigquery loaded: the table, the source files, the load job, and the release they hold.
Provisional, like PublishedDataset.
PullProgress
dataclass
¶
PullProgress(
stage: Literal["start", "copied"],
files_total: int,
bytes_total: int,
files_present: int = 0,
bytes_present: int = 0,
files_done: int = 0,
bytes_done: int = 0,
path: str | None = None,
file_bytes: int = 0,
)
How far pull_dataset or pull_to_bigquery has got, as passed to their progress callback.
Provisional, like PublishedDataset.
stage is "start" once the source files are listed and sized, before anything is copied or loaded, and
"copied" after each file pull_dataset commits; pull_to_bigquery reports "start" only, since its
one load job is atomic. Sizes are in bytes.
files_total
instance-attribute
¶
The number of Parquet files in the published dataset.
files_present
class-attribute
instance-attribute
¶
Of those, the files already at the destination with the source's size, which a resumed pull skips.
bytes_present
class-attribute
instance-attribute
¶
The total size of the files already present.
files_done
class-attribute
instance-attribute
¶
The files in place so far, those already present included.
bytes_done
class-attribute
instance-attribute
¶
The bytes in place so far, those already present included.
path
class-attribute
instance-attribute
¶
The file just copied, relative to the dataset's root; None at the start.
file_bytes
class-attribute
instance-attribute
¶
The size of the file just copied; 0 at the start.
PullSummary
dataclass
¶
PullSummary(
destination: str,
files: int,
bytes: int,
identity: AnnotationSourceIdentity,
source_uri: str,
skipped: int = 0,
)
What pull_dataset copied: the number of files and bytes, where from and to, and the release they hold.
Provisional, like PublishedDataset.
files and bytes count what this call copied. A resumed pull also counts the files it found already
complete and skipped (skipped), so files + skipped is the number of files in the copy.
JoinSpec
dataclass
¶
One LEFT JOIN the store adds to its materialize query to read an annotation source in SQL.
Adding this join lets the store read the source within its single query instead of calling annotate()
and merging in Python. For each spec the store builds:
`LEFT JOIN <relation> AS <alias> ON <alias>.<join_key> = <base_alias>.<join_key>`
and adds <alias>.<column> for each entry in columns to the materialized row.
relation is whatever goes between LEFT JOIN and AS. It is a backtick-quoted table reference for a
per-variant table, or a parenthesized subquery such as (SELECT variant_id, … FROM t GROUP BY
variant_id) when the source first collapses several per-transcript rows into one row per variant
(AlphaMissense and REVEL do this inside the source, not the store). A source that cannot fuse returns
None from sql_join, and the store reads it through annotate() instead.
SqlAnnotationSource
¶
Bases: AnnotationSource
An AnnotationSource that shares a SQL backend with the store and can fuse its read into the store's
materialize query.
A source subclasses this to signal that the store may read it inside one SQL query rather than through
annotate() and a Python merge. A SQL store finds these sources with
isinstance(source, SqlAnnotationSource). A plain AnnotationSource (Parquet, in-process, or a
different backend) does not subclass it, so the store reads that source through annotate(). Subclassing
is how a source opts in; the base AnnotationSource contract has no SQL methods.
sql_join
abstractmethod
¶
sql_join(ctx: SqlJoinContext) -> JoinSpec | None
Return a JoinSpec the store adds to its materialize query, or None if this source cannot fuse
for the given ctx. When it returns None, the store reads this source through annotate() instead.
SqlJoinContext
dataclass
¶
SqlJoinContext(
base_alias: str,
join_key: str = "variant_id",
variant_ids_param: str = "variant_ids",
variants_relation: str | None = None,
)
The information the store passes each source at materialize time so it can build a join (sql_join).
It holds only what the store's materialize step currently needs.
base_alias— the store's base relation, the scores side, which each source's join attaches to as the left side.join_key— the column both sides join on (variant_id).variant_ids_param— the name of the array query parameter the store binds to the job's variant ids (@variant_ids). A source whoserelationis an aggregating subquery, where AlphaMissense or REVEL collapse several per-transcript rows into one, uses this to restrict the large reference table to the job's variants before aggregating, with anINNER JOIN variantsscan. The outerLEFT JOINcannot do this, because it runs after theGROUP BYand so cannot prune the group's input.variants_relation— an optional SQL relation (a table id or subquery) that yields the job'svariant_ids. When set,{variant_filter}prunes withvariant_id IN (SELECT variant_id FROM <relation>)instead of the inlineUNNEST(@variant_ids)parameter. A job with millions of variants uses this to scope the query without exceeding BigQuery's request-size limit: it uploads a temp table and passes the reference here. The default,None, uses the inline-parameter prune, which suits small jobs and the emulator.
SqliteAnnotationSource
¶
SqliteAnnotationSource(
path: str = ":memory:",
columns: Sequence[AnnotationColumn] = (),
*,
name: str | None = None,
prioritize_predicate: Predicate | None = None,
identity: AnnotationSourceIdentity | None = None,
identity_scoped: bool = False,
)
Bases: AnnotationSource
Your own per-variant values, from a SQLite file. Build it in code with the columns the file holds.
It uses one table, variant_annotations, keyed by variant_id, with one column per AnnotationColumn. It
acts as a prioritizing source when you pass a prioritize_predicate, and a passive one otherwise. That
choice is configuration, not a separate subclass.
Unlike the read-only base class, this source can also be written to. Use add_annotations to load your
own reference data so it feeds into materialize, and get_unannotated_variants to find the variants it
has no row for yet.
identity= alone is metadata. With identity_scoped=True, the file holds the annotations of exactly one
AnnotationSourceIdentity, recorded in its _altar_annotation_identity table. A scoped source records its
identity the first time it opens a file with no annotation rows. Opening a file that records a different
identity raises AnnotationIdentityError, so a data release bump uses a new file rather than serving the
old release's rows. A file that already holds rows but records no identity was written before identities
were stored: a scoped source opens it, but refuses to read or write it until backfill_annotation_identity()
adopts its rows. Each operation re-reads the recorded identity, so an adoption by another instance is seen.
An unscoped source reads any file, but cannot write to one that records an identity.
backfill_annotation_identity
async
¶
backfill_annotation_identity() -> IdentityBackfill
Adopt a legacy file's rows as this source's identity, and return how many rows were adopted.
This is an explicit, reviewed one-time migration. It records the configured identity in a file that
holds annotation rows but no identity, asserting that the declared data release produced every row in
it. It needs identity_scoped=True. It returns IdentityBackfill(stamped=<rows adopted>, deleted=0):
the primary key already allows one row per variant, so nothing is deleted. A file that already records
this identity is left unchanged and stamped is 0, so a second call is a no-op; a file recording
another identity cannot be opened at all.
add_annotations
async
¶
add_annotations(rows: Tabular) -> InsertResult
Insert per-variant annotation rows. Each row is a mapping like
{"variant_id": ..., <column>: value, ...}. Rows use INSERT OR IGNORE on the variant_id primary
key, so a variant that already has a row is skipped rather than overwritten.
For an identity-scoped source, the rows belong to the file's identity, and a row whose
_altar_annotation_identity names another identity hash raises AnnotationIdentityError. An unscoped
source cannot write to a file that records an identity.
annotate
async
¶
Return {variant_id: {column: value}} for the requested variants that this source has rows for. A
variant with no row is left out of the result; the merge step then treats it as an empty annotation
(see merge_annotations).
get_unannotated_variants
async
¶
Return the variant_ids this file has no annotation row for, in input order.
For an identity-scoped source the file holds one identity (see the class docstring), so every stored row is a row of this source's identity.
close
¶
Close the source's SQLite connection; a second call does nothing.
Close a file-backed source when you are done with it, or open it in a with block, which closes it on exit.
VariantGeneLinkAnnotationSource
¶
VariantGeneLinkAnnotationSource(
store: VariantGeneLinkStore,
dataset: VariantGeneLinkDatasetRef,
*,
release: str,
genome_build: str = "hg38",
filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
prioritize_predicate: Predicate | None = None,
name: str | None = None,
column_prefix: str | None = None,
)
Bases: AnnotationSource
The genes each variant may regulate, from variant-gene links you import, such as ENCODE-rE2G.
It annotates variants with the org.kundajelab.altar.annotation.variant-gene-links columns from a link
store. A run spec declares it over a SQLite link store.
store is any VariantGeneLinkStore and dataset the published dataset in it to read; the in-memory and
SQLite stores, or a host's own, serve alike. release is the data release label of that dataset, which the
store does not record itself; it must change whenever the stored evidence changes. genome_build is spelled
as in score identity, "hg38" or "hg19" (supported_genome_builds), and selects the assembly the variant
IDs are on; a dataset that holds another assembly raises GenomeBuildMismatchError from annotate. filters
narrow every variant's query, for example to one biosample; because they change the values, the identity's
release becomes <release>+<fingerprint of the filters> when any is set. prioritize_predicate, evaluated over the
contract's columns such as link_top_score, makes the source gate prioritized; without one it only
annotates. name is the logical source name recorded in prioritized_source_names.
column_prefix lets two link datasets annotate one materialization, for example ENCODE-rE2G next to Open
Targets E2G. With a prefix such as "re2g", every declared column is re2g_<column> (re2g_link_top_score,
re2g_link_genes; the fields inside link_genes keep their names), labels name the prefix, and the default
name is re2g_variant_gene_links, so the sources' names and columns stay distinct. A predicate then reads
the prefixed columns. Without a prefix the columns are the contract's own. The prefix renames columns and
does not change their values, so identity() is the contract's either way. A model's dependency on the
contract reads the contract's column names, which only an unprefixed source declares.
Over a store, each requested variant is queried at its own locus through the store's public query contract,
every page of links is read, and the result is projected with project_variant_gene_links. A variant's locus
is the REF span of its minimal representation (minimal_variant_key), and its row is keyed by the ID as
passed. The locus includes the anchor base an indel's ID carries, so a deletion links to an element that ends
at its anchor base, although phyloP scores only the deleted bases. A variant on a chromosome the link contract
does not name, such as an unplaced contig, has no row. Store failures raise VariantGeneLinkSourceError; they
are never turned into an empty row.
One locus query per variant suits tests, notebooks and cohorts of up to some tens of thousands of variants.
For larger cohorts, write the links as a Parquet link dataset (write_link_dataset) and read it with
from_dataset, which reads many variants per query, locally or where the dataset is hosted, and projects the
same rows through the same function; its read failures raise VariantGeneLinkSourceError too. A link dataset
loaded into BigQuery is read with from_bigquery_table, one range-join query per chunk of variants.
release
property
¶
The identity's release: the dataset's release, plus a fingerprint of the filters when any is set.
from_dataset
classmethod
¶
from_dataset(
dataset: str | PathLike[str],
*,
storage_options: Mapping[str, Any] | None = None,
genome_build: str | None = None,
filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
prioritize_predicate: Predicate | None = None,
name: str | None = None,
column_prefix: str | None = None,
) -> Self
Read the Parquet link dataset write_link_dataset wrote at dataset, without a link store. Provisional.
dataset is a local path or a URI such as gs://... or hf://..., read in place, with
storage_options passed to its fsspec filesystem (see ParquetAllelicRecordBackend). The release, the
genome build and the VariantGeneLinkDatasetRef come from the metadata the writer records in every file
(LINK_DATASET_METADATA), read from the first file; a file that records none raises ValueError.
genome_build, when given, must be the build the metadata records, or GenomeBuildMismatchError is raised.
The other options are the constructor's; with filters the identity's release is the recorded release plus
the filters' fingerprint, as for a store.
annotate reads every requested variant's links by row group rather than one query per variant: the
files' footers, read once, say which row groups can hold a link overlapping each variant, and each such
row group is read once however many variants it serves, adjacent ones together; a remote dataset is read
with one ranged request per run of adjacent row groups. Its rows are projected through
project_variant_gene_links, as a store's are, so the two modes return the same rows for the same links.
from_bigquery_table
classmethod
¶
from_bigquery_table(
client: Any,
table_id: str,
*,
dataset: VariantGeneLinkDatasetRef,
release: str,
genome_build: str = "hg38",
filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
prioritize_predicate: Predicate | None = None,
name: str | None = None,
column_prefix: str | None = None,
) -> Self
Read a link dataset loaded into the BigQuery table table_id through client. Provisional.
table_id is a project.dataset.table holding a link dataset's columns, as altar pull --bigquery
loads one. A table holds no Parquet metadata, so dataset, release and genome_build say what it holds,
and the table's release labels are checked against them with one metadata call, no query
(check_bigquery_release): a table labelled with another release or build raises ValueError, and one
with no labels is read with a warning. The other options are the constructor's; with filters the
identity's release is release plus the filters' fingerprint, and the labels are checked against
release itself.
annotate runs one query per TABLE_QUERY_LOCI variants: their loci are bound as array query parameters
and range-joined against the table with the filters, and the links are projected through
project_variant_gene_links, so the rows are those of the Parquet and store modes. The source does not
own client; whoever made it closes it. A query BigQuery refuses raises VariantGeneLinkSourceError.
from_sqlite
classmethod
¶
from_sqlite(
path: str | PathLike[str],
dataset: VariantGeneLinkDatasetRef,
*,
release: str,
genome_build: str = "hg38",
filters: VariantGeneLinkFilters = VariantGeneLinkFilters(),
prioritize_predicate: Predicate | None = None,
name: str | None = None,
column_prefix: str | None = None,
) -> Self
Read dataset from the existing SqliteVariantGeneLinkStore file at path.
The file must already hold a published generation of dataset (see
SqliteVariantGeneLinkStore.open_published), so a mistyped path or dataset fails here instead of
annotating nothing. The source owns the store's connection, and close closes it.
from_options
classmethod
¶
Build the source over a SQLite link store from a run spec's options.
The options are sqlite_path, dataset as a {namespace, name, version} mapping, and release, plus
optionally genome_build, column_prefix, name, and filters, a mapping of VariantGeneLinkFilters
fields whose identifier filters are lists of strings. A predicate cannot be spelled in a spec, so a
spec-declared source only annotates; build the source in code to prioritize with it::
[sources.re2g]
type = "variant_gene_links"
[sources.re2g.options]
sqlite_path = "links.db"
dataset = { namespace = "encode", name = "re2g", version = "ENCFF229GDZ" }
release = "ENCFF229GDZ"
column_prefix = "re2g"
filters = { context_names = ["right kidney"] }
close
¶
Close what the source opened: the store from_sqlite opened, or the reader from_dataset opened.
A second call does nothing. A store passed to the constructor and a client passed to from_bigquery_table
stay open, because whoever created them closes them.
VariantGeneLinkFilters
dataclass
¶
VariantGeneLinkFilters(
context_ids: tuple[str, ...] = (),
context_names: tuple[str, ...] = (),
source_ids: tuple[str, ...] = (),
gene_ids: tuple[str, ...] = (),
methods: tuple[str, ...] = (),
data_releases: tuple[str, ...] = (),
minimum_score: float | None = None,
maximum_score: float | None = None,
)
Relation-level filters applied to every variant's link query, with VariantGeneLinkQuery semantics.
Context IDs and names select their union; the other identifier tuples and the inclusive score bounds are
intersections. Values are stripped and deduplicated. The filters change which links a variant has, so
VariantGeneLinkAnnotationSource folds them into its identity's release.
BiosampleContext
dataclass
¶
The cell, tissue, or other biosample in which a link was measured or predicted.
GenomeBuild
¶
Bases: StrEnum
Genome assemblies supported by the normalized link contract.
GenomeBuildMismatchError
¶
Bases: VariantGeneLinkQueryError
The query and source use different genome assemblies.
GenomicInterval
dataclass
¶
GenomicInterval(
chromosome: str,
start: int,
end: int,
genome_build: GenomeBuild,
coordinate_system: CoordinateSystem = "0-based-half-open",
)
A normalized 0-based, half-open genomic interval [start, end).
overlaps
¶
overlaps(other: GenomicInterval) -> bool
Return whether two same-build intervals overlap.
InvalidCursorError
¶
Bases: VariantGeneLinkQueryError
A pagination cursor is malformed or belongs to a different query.
LinkDistances
dataclass
¶
LinkDistances(
element_to_gene_bp: float | None = None,
variant_to_gene_bp: float | None = None,
variant_to_element_bp: float | None = None,
interpretation: str | None = None,
)
Distances retained when a source provides them, in base pairs.
LinkProvenance
dataclass
¶
LinkProvenance(
source_record_id: str,
source_dataset: str,
source_url: str | None = None,
study_id: str | None = None,
publication_ids: tuple[str, ...] = (),
metadata: Mapping[str, Any] = dict(),
)
Evidence identifiers and source-specific metadata retained without flattening.
LinkScore
dataclass
¶
LinkScore(
value: float | None,
direction: ScoreDirection,
interpretation: str,
threshold: float | None = None,
threshold_rule: ThresholdRule | None = None,
threshold_applied: bool | None = None,
threshold_interpretation: str | None = None,
)
A source score without conflating a missing value with numeric zero.
RegulatoryElement
dataclass
¶
RegulatoryElement(
id: str,
interval: GenomicInterval,
element_type: str | None = None,
)
Identity and genomic extent of one regulatory element.
ScoreDirection
¶
Bases: StrEnum
How to interpret increasing values of a source's link score.
TargetGene
dataclass
¶
TargetGene(
id: str,
namespace: str,
namespace_version: str | None = None,
symbol: str | None = None,
)
A target gene identifier with an explicit namespace and optional release/symbol.
VariantGeneLink
dataclass
¶
VariantGeneLink(
record_id: str,
locus: VariantLocus,
regulatory_element: RegulatoryElement | None,
target_gene: TargetGene,
context: BiosampleContext | None,
source_id: str,
source_name: str,
method: str,
data_release: str,
score: LinkScore,
distances: LinkDistances,
provenance: LinkProvenance,
)
One loss-minimizing variant/locus-to-gene evidence record.
regulatory_element is present for mediated evidence such as enhancer-to-gene links. It is absent for
sources that directly associate a locus or allele with a gene. Consumers that specifically project an
enhancer graph must reject direct records instead of inventing a regulatory element.
VariantGeneLinkDataError
¶
Bases: VariantGeneLinkSourceError
The source returned malformed, unstable, or ambiguously duplicated data.
VariantGeneLinkError
¶
Bases: Exception
Base exception for variant-to-gene link operations.
VariantGeneLinkPage
dataclass
¶
VariantGeneLinkPage(
links: tuple[VariantGeneLink, ...],
next_cursor: str | None = None,
)
A bounded page of links; an empty tuple with no cursor is a successful empty result.
VariantGeneLinkQuery
dataclass
¶
VariantGeneLinkQuery(
locus: VariantLocus,
context_ids: tuple[str, ...] = (),
context_names: tuple[str, ...] = (),
source_ids: tuple[str, ...] = (),
gene_ids: tuple[str, ...] = (),
methods: tuple[str, ...] = (),
data_releases: tuple[str, ...] = (),
minimum_score: float | None = None,
maximum_score: float | None = None,
page_size: int = 100,
cursor: str | None = None,
)
One interval-overlap query plus relation-level filters and a bounded page size.
Context IDs and names are alternative selectors (their union). Other non-empty identifier tuples are intersections, as are the inclusive score bounds. No score cutoff is applied when both bounds are absent.
VariantGeneLinkQueryError
¶
Bases: VariantGeneLinkError, ValueError
The normalized query is invalid for the selected source.
VariantGeneLinkSource
¶
Bases: ABC
Queryable one-to-many variant/locus-to-gene evidence.
A valid query with no matching links returns VariantGeneLinkPage(links=(), next_cursor=None).
Network, storage, malformed-source-data, and pagination failures raise VariantGeneLinkSourceError
(or a subclass) and must never be translated into an empty page.
query_links
abstractmethod
async
¶
query_links(
query: VariantGeneLinkQuery,
) -> VariantGeneLinkPage
Return one stable, bounded page of links for query.
stream_links
async
¶
stream_links(
query: VariantGeneLinkQuery,
) -> AsyncIterator[VariantGeneLink]
Stream all pages without retaining the full relation in memory.
VariantGeneLinkSourceError
¶
Bases: VariantGeneLinkError
A source could not answer a valid query because of network, storage, or source-data failure.
VariantLocus
dataclass
¶
VariantLocus(
interval: GenomicInterval,
reference_allele: str | None = None,
alternate_allele: str | None = None,
)
A normalized query locus, optionally carrying normalized alleles.
interval always uses 0-based, half-open coordinates. from_vcf is the explicit conversion from
a 1-based VCF/Open Targets variant position; the reference allele determines the reference span.
from_vcf
classmethod
¶
from_vcf(
*,
chromosome: str,
position: int,
reference_allele: str,
alternate_allele: str,
genome_build: GenomeBuild,
) -> VariantLocus
Convert a 1-based VCF locus into the contract's 0-based, half-open interval.
The interval is the REF span, so an indel's locus includes the VCF anchor base before the inserted or deleted bases.
InMemoryVariantGeneLinkStore
¶
Deterministic reference store used by examples and Cartesian conformance tests.
LinkRecordWriteResult
dataclass
¶
Counts returned by an idempotent generation write.
SqliteVariantGeneLinkStore
¶
Portable SQLite store for canonical evidence with indexed overlap and relation filters.
open_published
classmethod
¶
open_published(
path: str | PathLike[str],
dataset: VariantGeneLinkDatasetRef,
) -> Self
Open an existing store file to read dataset, which must have at least one published generation.
The constructor creates a missing file and its schema, which suits an importer. A reader configured from
a path, such as a run spec's link annotation source, uses this instead, so a mistyped path or dataset
fails here rather than reading as a dataset without links. Raises FileNotFoundError for a missing file
and ValueError when the file has no published generation of dataset.
StoredVariantGeneLinkSource
¶
StoredVariantGeneLinkSource(
store: VariantGeneLinkStore,
dataset: VariantGeneLinkDatasetRef,
*,
name: str,
binding_id: str,
binding_version: str,
failure_label: str = "variant-gene link store",
)
Bases: VariantGeneLinkSource
Reusable public source over a generic canonical evidence store.
decode_record
¶
decode_record(
record: VariantGeneEvidenceRecord, locus: VariantLocus
) -> VariantGeneLink
Project a canonical record into a result; bindings may restore typed source metadata.
VariantGeneEvidenceRecord
dataclass
¶
VariantGeneEvidenceRecord(
record_id: str,
evidence_interval: GenomicInterval,
regulatory_element: RegulatoryElement | None,
target_gene: TargetGene,
context: BiosampleContext | None,
source_id: str,
source_name: str,
method: str,
data_release: str,
score: LinkScore,
distances: LinkDistances,
provenance: LinkProvenance,
metadata: Mapping[str, JsonValue] = dict(),
)
One durable, query-independent locus/element-to-gene evidence record.
to_link
¶
to_link(locus: VariantLocus) -> VariantGeneLink
Attach a caller's overlapping locus and return the public query result.
VariantGeneLinkDatasetRef
dataclass
¶
Stable identity for one independently versioned collection of canonical link evidence.
key
property
¶
Return a deterministic store key without assigning semantics to separators in components.
VariantGeneLinkStore
¶
Bases: Protocol
Persist canonical evidence independently of its scientific source format.
Records are written into immutable generations and become queryable only after publish_generation.
This lets importers checkpoint bounded batches without exposing partial releases.
write_records
async
¶
write_records(
dataset: VariantGeneLinkDatasetRef,
generation_id: str,
records: Sequence[VariantGeneEvidenceRecord],
) -> LinkRecordWriteResult
Idempotently stage a bounded batch in generation_id.
publish_generation
async
¶
publish_generation(
dataset: VariantGeneLinkDatasetRef, generation_id: str
) -> None
Atomically make all records in a staged generation visible to readers.
generation_size
async
¶
generation_size(
dataset: VariantGeneLinkDatasetRef, generation_id: str
) -> int
Return the number of unique staged or published records in a generation.
available_builds
async
¶
available_builds(
dataset: VariantGeneLinkDatasetRef,
) -> Sequence[GenomeBuild]
Return genome builds represented by published generations.
fetch_page
async
¶
fetch_page(
dataset: VariantGeneLinkDatasetRef,
query: VariantGeneLinkStoreQuery,
) -> Sequence[VariantGeneEvidenceRecord]
Return published records ordered strictly by record_id.
VariantGeneLinkStoreQuery
dataclass
¶
VariantGeneLinkStoreQuery(
interval: GenomicInterval,
context_ids: tuple[str, ...] = (),
context_names: tuple[str, ...] = (),
source_ids: tuple[str, ...] = (),
gene_ids: tuple[str, ...] = (),
methods: tuple[str, ...] = (),
data_releases: tuple[str, ...] = (),
minimum_score: float | None = None,
maximum_score: float | None = None,
after_record_id: str | None = None,
limit: int = 101,
)
Backend-neutral indexed request derived from a public link query.
from_query
classmethod
¶
from_query(
query: VariantGeneLinkQuery,
) -> VariantGeneLinkStoreQuery
Translate the public query while decoding its opaque cursor once in the source layer.
canonical_chromosome
¶
Return the canonical chr-prefixed spelling used in portable keys.
Primary chromosomes accept common aliases (1/chr1/CHR01 and
M/MT/chrM) in any case. Other reference contigs keep their exact
spelling after an optional case-insensitive chr prefix is normalized, so
CHRUn_KI270302v1 becomes chrUn_KI270302v1 but chrun_KI270302v1
stays as written. Non-ASCII text, whitespace inside the name, and characters
outside the VCF contig-name set (including :) raise
VariantIdentityError. This is a naming rule only; it does not claim that
contigs from different assemblies are interchangeable.
canonical_variant_id
¶
canonical_variant_id(
chromosome: str,
position: int,
reference_allele: str,
alternate_allele: str,
) -> str
Return the canonical text key for decomposed one-based locus fields.
minimal_representation
¶
minimal_representation(
position: int,
reference_allele: str,
alternate_allele: str,
) -> tuple[int, str, str]
Trim the bases that REF and ALT share, keeping at least one base in each.
The shared suffix is removed first, then the shared prefix, and position
advances by one for each prefix base removed, so the result names the same
change at the same locus: (10, "CAT", "CT") becomes (10, "CA", "C")
and (10, "TCA", "TGA") becomes (11, "C", "G"). Trimming the suffix
first leaves an indel anchored on the base before it, the VCF convention. A
SNV, and alleles that share no end base (A/AT, AT/A), come
back unchanged. Alleles are spelled as the key spells them: trimmed of ASCII
whitespace and uppercased.
The alleles must have a key: each is [ACGTN]+ and they differ, like
VariantKey. Symbolic (<DEL>), spanning-deletion *, breakend and
missing alleles raise VariantIdentityError rather than being trimmed as if
their characters were bases, and so do equal alleles, which would otherwise
collapse to one shared base that is not a variant. The position is not
validated beyond being an int, so a one-based or zero-based position
stays in its own convention.
Trimming is a pure function of the alleles. Left alignment, which shifts an indel through a repeat using the reference sequence, is a separate step that ingestion applies around this one.
allelic_key
¶
allelic_key(variant_id: str) -> AllelicKey
Return the backend key for a canonical chr:pos:ref:alt variant ID.
The key's variant_id is the ID as passed. Its other fields are the parts of minimal_variant_key: a
chr-prefixed chromosome, the one-based position, and the trimmed reference and alternate alleles. A
malformed or non-canonical ID raises VariantIdentityError (a ValueError). A collection that spells
chromosomes another way, such as 1 rather than chr1, can rewrite the key's chromosome with
dataclasses.replace; the key keeps the caller's variant_id either way.
Provisional: the helper is expected to change.
minimal_variant_key
¶
minimal_variant_key(variant_id: str) -> VariantKey
Return the key an annotation lookup reads for a canonical chr:pos:ref:alt variant ID.
The ID is checked with VariantKey.require_canonical, which raises VariantIdentityError (a ValueError)
for a malformed or non-canonical one. Its alleles are then trimmed to their minimal representation
(minimal_representation), so chr17:43124027:ACTC:AC reads as chr17:43124027:ACT:A. Trimming needs no
reference. Left alignment does, so an indel written at the right end of a repeat keeps its position.
allelic_key, PositionScoreAnnotationSource and VariantGeneLinkAnnotationSource look variants up through
this function and key their rows by the ID the caller passed.
Provisional: the helper is expected to change.
applicable_annotation_sources
¶
applicable_annotation_sources(
models: Sequence[ScoredModel],
sources: Sequence[AnnotationSource],
) -> tuple[AnnotationSource, ...]
Return the sources that may annotate the models' variants, in configuration order.
The variants' assembly is the genome build the models' run identities record. A source whose identity()
declares another assembly is dropped. A source that declares no identity serves the default assembly
(DEFAULT_GENOME_BUILD, hg38): it is kept for hg38 variants and dropped for variants on any other assembly,
whether or not Altar knows its label. With no model recording a genome build, every source is kept. Models
scored on different assemblies raise AnnotationGenomeBuildError when any source is configured, because no
single assembly then decides which sources apply.
Every store's materialize and export call this and join only the returned sources. It never raises for a
dropped source; resolve_annotation_dependencies decides whether a drop is a configuration error.
resolve_annotation_dependencies
¶
resolve_annotation_dependencies(
models: Sequence[ScoredModel],
sources: Sequence[AnnotationSource],
*,
contracts: Sequence[AnnotationContract] = (),
) -> tuple[ResolvedAnnotationDependency, ...]
Check that sources satisfy every annotation dependency of models, and return how.
For each AnnotationDependency in each model's manifest.prioritization:
- If a source's
identity().source_idis the dependency'ssource_id, that source must implement a compatible contract version (same major version, and a version no older than the dependency's) and declare every dependency column. Each of those columns must have the logical type (dtype,repeatedand nestedfields; labels may differ) that every known contract with the dependency'ssource_idand major version gives it. OtherwiseAnnotationDependencyErroris raised. - If no source declares that contract but every dependency column is declared by some source, the
dependency resolves by column name and an
UndeclaredAnnotationContractWarningis emitted. - Otherwise
AnnotationDependencyErroris raised naming the contract and the missing columns.
Only the sources applicable_annotation_sources keeps take part. A dependency they do not satisfy is:
- unavailable, when source classes implement the dependency's contract and none of them serves the variants'
assembly. It resolves with the missing columns in
unavailable, and oneAnnotationUnavailableWarningper contract names the models; AnnotationGenomeBuildErrorotherwise, when a dropped source would have supplied it (it declares the contract or supplies the missing columns): a source on the variants' assembly could exist, so the dropped one is a configuration error;AnnotationDependencyErrorotherwise, as in step 3.
The classes are the healthy altar.annotation_sources entries, plus the classes of the configured sources,
whose annotation_contract() names the contract. A class serves an assembly when its
supported_genome_builds contains it or is None. A contract no class implements could be served on any
assembly. Whether a mismatch warns or raises therefore depends on which source packages are installed. A
dropped source no dependency needed is named in one AnnotationUnavailableWarning. The UCSC and GRC
spellings of one assembly, such as hg38 and GRCh38, match. With no model recording a genome build, every
source that declares an identity must declare the same assembly; otherwise AnnotationGenomeBuildError is
raised.
The known contracts are the ones Altar publishes, such as REGION_ANNOTATION_CONTRACT, plus contracts. A
dependency on a contract the resolver does not know, such as a binding's or a host's own, gets the name check
only. Pass that contract in contracts to check its column types too, or run
altar.testing.assert_implements_contract to check a source against its whole contract.
Store materialize methods, and the BigQuery exports that compute prioritized, call this once per call,
without contracts. A host can also call it up front, with its own contracts, to check a configuration
before submitting work.
unavailable_annotation_columns
¶
unavailable_annotation_columns(
resolved: Iterable[ResolvedAnnotationDependency],
) -> frozenset[str]
Return every column the resolved dependencies mark unavailable; a store evaluates each as missing (null).
fetch_annotations
async
¶
fetch_annotations(
sources: Sequence[AnnotationSource],
variant_ids: Sequence[str],
) -> dict[str, dict[str, Any]]
Fetch each source's annotations for variant_ids and merge them with merge_annotations.
This builds the annotation input to materialize_variant from the configured annotation sources.
Sources are fetched concurrently and merged in declaration order. Their declared column names must be
globally unique, and the sources that declare an identity() must declare one genome build, as in
materialization. Either problem raises before any source is called: a collision SchemaCollisionError, a
second build AnnotationGenomeBuildError. Each annotate must return the portable
variant_id -> {column: value} mapping.
Each ID must be canonically spelled (VariantKey.require_canonical); a non-canonical one raises
VariantIdentityError. The result is keyed by the IDs as passed. The shipped allelic, position and
variant-gene link sources look each ID up at its minimal representation (minimal_variant_key), so
chr17:43124027:ACTC:AC finds the row stored for chr17:43124027:ACT:A. A writable lake, such as
SqliteAnnotationSource, stores and reads rows under the ID as written. Left alignment is the caller's job:
no source left-aligns, so left-align indels against the reference first (altar.variants.left_align), or an
indel written at the right end of a repeat finds no row.
get_annotation_source_registry
¶
get_annotation_source_registry() -> PluginRegistry[
type[AnnotationSource]
]
Return the shared registry of annotation sources (the altar.annotation_sources group).
load_annotation_identity
¶
load_annotation_identity(
sidecar: str
| bytes
| PathLike[str]
| Mapping[str, Any],
) -> AnnotationSourceIdentity
Read the AnnotationSourceIdentity recorded in an annotation identity sidecar.
The region annotate command writes the sidecar as <out>.identity.json next to its JSONL rows. It is a JSON
object with "format": 1, the identity's four fields under "identity", and their "identity_hash"; other
keys, such as columns, provenance and row_count, are informational and ignored here.
sidecar is the sidecar's JSON text (str or bytes), its already-parsed object, or a local file path
(os.PathLike, such as a pathlib.Path; a plain str is always JSON text). Altar never fetches the sidecar
from object storage: a host reads the bytes with its own client and passes them here, then passes the result
as expected_identity to BigQueryAnnotationSource.add_annotations_from_uri.
Raises AnnotationIdentityError when the sidecar is not a JSON object, has another format, lacks
identity, or records an identity_hash that the identity does not produce, and ManifestError when the
identity fields themselves are invalid.
merge_annotations
¶
merge_annotations(
variant_ids: Sequence[str],
tables: Sequence[AnnotationTable],
*,
source_names: Sequence[str] | None = None,
) -> dict[str, dict[str, Any]]
Left-merge several sources' annotation tables into one mapping per variant.
The left side is variant_ids, the variants being scored. Every id gets an entry, even one that no source
annotated, which yields {}. Each table is the portable variant_id -> {column: value} shape a source
returns from annotate. Two sources may not supply the same column: doing so is ambiguous even when
their declared dtypes agree, because their values may differ. A row a source returns for a variant not
in variant_ids is dropped.
This is the portable join that builds the annotation argument to materialize_variant. The BigQuery
store instead fuses its annotation reads into one query through SqlAnnotationSource.sql_join.
staged_annotation_source
async
¶
staged_annotation_source(
source: AnnotationSource,
client: Any,
*,
staging_dataset: str,
variants_relation: str | None = None,
variant_ids: Sequence[str] | None = None,
coverage: BigQueryAllelicRecordBackend | None = None,
batch_size: int = DEFAULT_STAGE_BATCH_SIZE,
expiration: timedelta | None = DEFAULT_STAGE_EXPIRATION,
) -> AsyncIterator[BigQueryAnnotationSource]
Stage source's annotations for one job in a scratch table and yield a SQL source bound to it.
source is a source without SQL of its own. The yielded source has the same name, columns, identity, and
prioritization rule, and reads the scratch table. Its joins are limited to each store query's variants.
Give the job's variants as exactly one of variant_ids or variants_relation, a SQL relation with a
variant_id column such as a backtick-quoted table id. With variants_relation and a coverage backend,
one query finds the variants that have records and reads those records. Only those variants are then
annotated. Without coverage, every variant id in the relation is read into Python and annotated.
staging_dataset is the project.dataset that holds scratch tables. Each table expires after
expiration as a backstop. The table is dropped when the block exits, including on error or
cancellation. With expiration=None a table left behind by a crashed process is never removed.
Raises ValueError if source returns a variant it was not asked for, and SchemaCollisionError if it
returns undeclared columns.
link_dataset_release_fields
¶
link_dataset_release_fields(
*,
release: str,
genome_build: str,
dataset: VariantGeneLinkDatasetRef,
) -> dict[str, str]
Return the footer fields that identify a link dataset's release, without their prefix. Provisional.
genome_build is Altar's spelling (hg38 or hg19), as PublishedDataset requires; the rows keep the link
contract's spelling. dataset is recorded as its namespace, name and version.
link_dataset_row
¶
link_dataset_row(
record: VariantGeneEvidenceRecord,
) -> dict[str, Any]
Return the link dataset row of one stored evidence record: its interval and its link_genes record.
Provisional, like write_link_dataset.
write_link_dataset
¶
write_link_dataset(
rows: Iterable[Mapping[str, Any] | RecordBatch | Table],
output: str | PathLike[str],
*,
release: str,
genome_build: str,
dataset: VariantGeneLinkDatasetRef,
overwrite: bool = False,
temp_directory: str | PathLike[str] | None = None,
) -> LinkDatasetSummary
Write link rows as a link dataset in the local directory output. Provisional.
rows yields link rows as mappings (link_dataset_row makes one from an evidence record) or as Arrow record
batches or tables, in any mix and any order; each has exactly the layout's columns: chromosome, start,
end, genome_build and the LINK_GENE_FIELDS. A chromosome is in the link contract's spelling (7, not
chr7) and genome_build is the contract's name for genome_build (GRCh38 for hg38). The rows are
consumed once, in batches, so a build of tens of millions of links streams.
The rows are staged in a DuckDB database under temp_directory (by default the system's), which also takes
the sort's spill, so memory stays bounded. A row repeated exactly is written once and counted in duplicates;
a record_id on two different rows raises ValueError, as does a dataset with no rows or a row that breaks
the layout. The rows are then written sorted by chromosome, start, end and record_id, one file per
chromosome under chromosome=<value>/, in row groups of 32,000 rows, each file recording
link_dataset_release_fields through LINK_DATASET_METADATA and written under a .partial name until it is
complete.
output must be empty or absent; with overwrite, a previous link dataset there is replaced. A failed write
leaves nothing behind. Write locally, then copy the directory to a bucket.
add_release_argument
¶
add_release_argument(
parser: ArgumentParser,
*,
help: str,
deprecated: Sequence[str] = (),
required: bool = True,
) -> None
Add --release, the upstream data release a builder reads, to parser; its value is args.release.
help is the option's help text. Each spelling in deprecated, such as "--version", sets the same value
and is left out of --help. Using one prints warning: <spelling> is deprecated; use --release to
standard error. Giving two different values is a usage error.
build_inputs
¶
build_inputs(
input_paths: Sequence[str | PathLike[str]],
stdin_name: str | None = None,
*,
connection: Connection | None = None,
) -> list[BuildInput]
Resolve a builder's inputs: local files, URIs, and standard input once as STDIN, named stdin_name.
A URI such as gs://bucket/chr1.vcf.bgz or https://host/chr1.vcf.gz is read in place through fsspec,
opened with connection.storage_options(uri), or with the provider's ambient credentials when there is no
connection; its name is the last segment of its path. A file:// URI is a local file. A missing fsspec or
provider raises an ImportError that names the Altar extra to install.
Raises ValueError when there is no input, standard input is listed twice, stdin_name is given without
STDIN or is not a bare file name, or two inputs would write the same output name; and FileNotFoundError
when a file or a URI does not exist.
build_output
¶
build_output(
output: Path,
*,
overwrite: bool,
partition_column: str,
label: str,
) -> Iterator[Path]
Make output an empty directory for the body of the with, and delete what the body wrote if it fails.
The directory must be empty or absent. With overwrite, a previous build there, <partition_column>=*
directories of Parquet files and nothing else, is deleted first; anything else is refused, naming label as
the build that would not have written it. If the body raises, the output is removed when this call created
it, and emptied otherwise.
canonical_chromosome_runs
¶
Split a batch of records into runs on one chromosome: (canonical chromosome, start, stop) row ranges.
Each spelling is canonicalized once with canonical_chromosome, so 1 and chr1 side by side are one run,
and an invalid name raises its VariantIdentityError. Sorted input has one run per chromosome in a batch;
feed each run's positions to RecordOrder.advance_batch. Provisional, like BuildOutput.
check_unique_keys
¶
check_unique_keys(
output: Path,
*,
unique_key: Sequence[str] = (VARIANT_ID_COLUMN,),
) -> None
Reject a unique_key written twice in a partition of output, across inputs or by trimming.
unique_key is variant_id alone for a dataset of one row per allele, or variant_id and the columns that
tell an allele's records apart, such as a transcript. Keys never span chromosomes, so each
<partition_column>=* directory is checked on its own.
minimal_representations
¶
minimal_representations(
positions: Array, references: Array, alternates: Array
) -> tuple[Array, Array, Array]
Trim a batch of alleles as minimal_representation trims each: return their positions, REFs and ALTs.
The alleles must already have a key ([ACGTN]+, uppercase, REF and ALT different). Only alleles whose REF and
ALT both have more than one base can share a base to trim, so the others are returned as they are and only
those are trimmed, one at a time. Provisional, like BuildOutput.
require_rows
¶
Reject a build that wrote no rows: an empty dataset is a wrong input, not an annotation of nothing.
read says what the build consumed, such as "3 records read, 3 skipped", and hint what to check.
storage_option_connection
¶
storage_option_connection(
pairs: Sequence[str],
) -> Connection | None
Return a connection with the storage options a builder's KEY=VALUE arguments give, or None for none.
A value that reads as JSON other than a string takes that type, so anon=true is a boolean and
target_options={"auto_mkdir": true} a mapping; any other value is a string. Credentials never come from a command line: a key that would carry one,
such as secret or token (other than gcsfs's token=anon and its other credential modes), is refused,
and the provider's ambient credentials are used instead. Provisional, like BuildOutput.
position_score_release_fields
¶
position_score_release_fields(
*,
contract: AnnotationContract,
release: str,
genome_build: str,
) -> dict[str, str]
Return the footer fields that identify a position score dataset's release, without their prefix. Provisional.
contract is the track's contract, which must declare exactly one float column; its source_id is recorded as
the track, so a dataset built for one contract is never read under another. genome_build is hg38 or
hg19; an alias such as GRCh38 is recorded in Altar's spelling.
bigquery_release_labels
¶
bigquery_release_labels(
identity: AnnotationSourceIdentity,
) -> dict[str, str]
Return the BigQuery table labels that record the release identity names.
Provisional, like PublishedDataset.
Three labels record the contract's source_id (altar_source), the release (altar_release) and the
genome_build (altar_genome_build). A label value may hold only lowercase letters, digits, _ and
-, at most 63 of them, so each value is the identity field lowercased, with every other character, such
as ., replaced by _: release 4.1.1-genomes is 4_1_1-genomes, and source
org.kundajelab.altar.annotation.gnomad is org_kundajelab_altar_annotation_gnomad. A longer value
keeps its first 54 characters, then - and the first 8 hex digits of the SHA-256 of the original text.
The encoding is not reversible, so labels are compared, never decoded: check_bigquery_release encodes
the identity a source expects and compares the result. The table's description names the release in full.
check_bigquery_release
¶
check_bigquery_release(
client: Any,
table: str,
identity: AnnotationSourceIdentity,
) -> bool
Check that the BigQuery table table holds the release identity names, as far as its labels say.
Provisional, like PublishedDataset.
Reads the table's labels with one metadata request (client.get_table): no query runs and no bytes are
billed. A table whose Altar labels (bigquery_release_labels) name another source, release or genome build
raises ValueError, and one with all three matching returns True. A table with no Altar label, such as
one loaded by hand rather than by pull_to_bigquery, is accepted with a UserWarning naming the release
it is assumed to hold, and False is returned. A client without get_table, such as a test double,
cannot be checked and also returns False. An error looking the table up, such as a missing table,
propagates.
The labels say which release was loaded, not that the rows were never edited since; the check is a cheap guard against reading one release under another's identity.
pull_dataset
¶
pull_dataset(
published: PublishedDataset,
destination: str | PathLike[str],
*,
location: str | None = None,
source_connection: Connection | None = None,
destination_connection: Connection | None = None,
overwrite: bool = False,
progress: Callable[[PullProgress], None] | None = None,
) -> PullSummary
Copy every Parquet file of published to destination, keeping their paths, and check the copy's release.
Provisional, like PublishedDataset.
The files are read from the published location named location, such as "hf" for a Hugging Face mirror,
or from the primary location by default (PublishedDataset.location); every location holds the same files, so
the copy is the same whichever is read. An unknown name raises ValueError naming the dataset's locations.
destination is a local directory or a URI such as gs://my-bucket/gnomad-4.1.1-genomes; the files keep
their paths relative to the location's URI, so the copy has the same <partition>=<chromosome>/ layout and
a source reads it as an owned dataset. The published files are read anonymously, or with the storage options
of source_connection, such as one that bills a requester-pays read. The copy is written with the storage
options of destination_connection, or with the provider's ambient credentials when none is given.
Files are streamed one at a time, in chunks, never held whole in memory. Each is written under a temporary name and committed when complete: renamed into place on a local disk, finalized on an object store, which makes an object visible only once its upload completes. A reader of the destination therefore never sees a partial file, and a failed copy leaves at most the files committed before it.
A pull resumes: the destination may be absent, empty, or hold files of this dataset, such as those an
interrupted pull committed. A file whose path is already there with the source's size is skipped, and every
other file is copied; a temporary file a killed pull left beside one of the dataset's files is removed. A
destination holding any other file is refused, unless overwrite is set; then every file is copied again,
replacing files of the same path, and nothing else is deleted. A Parquet file that is not part of the dataset
is refused either way, before anything is copied, because a source would read it too.
After copying, every Parquet file under destination must be one of the dataset's, and every one, copied or
skipped, must record the published release and genome build (PublishedDataset.check_release); otherwise
ValueError is raised, naming what to remove or rebuild. A skipped file of the same size but another
release therefore fails the pull rather than passing as this one.
progress, when given, is called with a PullProgress once the files are listed and sized ("start",
which counts the files already present) and after each file is copied ("copied").
Raises FileExistsError for a destination holding other files without overwrite, FileNotFoundError
when the published dataset lists no Parquet file, and ValueError when published declares no release
metadata, so the copy could not be checked. Returns the number of files and bytes copied, the number of
files skipped, and the URI they came from.
pull_to_bigquery
¶
pull_to_bigquery(
published: PublishedDataset,
table: str,
*,
connection: Connection,
location: str | None = None,
source_uri: str | None = None,
source_connection: Connection | None = None,
overwrite: bool = False,
cluster_by: Sequence[str] | None = None,
progress: Callable[[PullProgress], None] | None = None,
) -> BigQueryLoadSummary
Load the Parquet files of published into the owned BigQuery table table with one load job.
Provisional, like PublishedDataset.
table is a project.dataset.table reference in a dataset that exists. The load job runs and is billed
through connection.bigquery_client(), as the connection's identity. BigQuery loads only from Cloud
Storage, so the files are read from the published location named location (the primary by default), which
must then be a gs:// URI, or from source_uri, a gs:// copy made with pull_dataset, such as one in
the table's region or one pulled from a mirror that is not on Cloud Storage. location and source_uri are
exclusive. BigQuery reads the files itself, and the destination dataset's location must be one that can load
from the bucket's.
Before loading, the release metadata of every source file is checked against the published release, read
with source_connection (or anonymously) for the published URI and with connection for source_uri. A
table holds no Parquet metadata, so after the load the release is written as table labels
(bigquery_release_labels), which a source reading the table checks (check_bigquery_release), and the
table's description names it in full. Labelling is one metadata update; if it fails, the loaded table is
kept, a warning says so, and the summary's labels is empty.
The load takes <uri>/*.parquet (or the one file a .parquet URI names), so every file under the
dataset is loaded, whatever its partition directory. Hive partitioning is not used: every file already holds
its partition column (chr in the builders' layouts), and BigQuery refuses partition keys that are also
columns of the files. The table is clustered on cluster_by, by default the dataset's own
(PublishedDataset.cluster_by): variant_id, which every allele-keyed builder writes, or chromosome, start
for a link dataset; pass () for none. The load infers Parquet lists, so a repeated column becomes an
ARRAY, and a list of records an ARRAY<STRUCT<...>>, as a binding's SQL relation reads it; without list
inference,
BigQuery would keep Parquet's encoding, STRUCT<list ARRAY<STRUCT<element ...>>>.
The table must not exist or be empty (WRITE_EMPTY); overwrite replaces its rows (WRITE_TRUNCATE).
A load job is atomic, so an interrupted pull loads nothing and is simply run again. progress, when given, is
called once with a "start" PullProgress that counts and sizes the source files, before they are checked
and loaded. Waits for the load job and returns its summary. Without the altar[bigquery] extra, raises an
ImportError that names it.
owned_dataset_storage_options
¶
owned_dataset_storage_options(
label: str,
dataset: str | PathLike[str],
*,
storage_options: Mapping[str, Any] | None = None,
connection: Connection | None = None,
) -> dict[str, Any] | None
Return the fsspec storage options that read an owned dataset at dataset, a path or a URI.
With a connection, they are the connection's options for dataset's URI; a connection returns none for a
local path. Otherwise they are storage_options, or None when that is not given, and the provider's ambient
credentials apply. label names the data in the error raised when both are given: a connection supplies the
storage options, so a source takes one or the other. A connection that cannot reach dataset's scheme raises
its CapabilityNotSupported.
unknown_options_message
¶
unknown_options_message(
label: str,
from_options: Callable[..., object],
unknown: Iterable[str],
*,
code_only: Sequence[str] = (),
) -> str
Return the error message for run spec options that from_options does not take.
The message names the unknown options, sorted, and lists the keyword-only parameters of from_options in
their declared order. label names the data, such as "AlphaMissense". Parameters named in code_only
take Python objects a run spec cannot spell: they are left out of the list and named in a closing clause.
A source raises ValueError with it::
if unknown:
raise ValueError(unknown_options_message("AlphaMissense", cls.from_options, unknown))
link_gene_record
¶
link_gene_record(
link: VariantGeneLink | VariantGeneEvidenceRecord,
) -> dict[str, Any]
Return one link_genes record: the LINK_GENE_FIELDS of link, with None for what it lacks.
link is a query's VariantGeneLink or the stored VariantGeneEvidenceRecord it was made from; both carry
the same fields.
link_genome_build
¶
link_genome_build(genome_build: str) -> GenomeBuild
Return the link contract's GenomeBuild for a score-identity build name such as "hg38".
project_variant_gene_links
¶
project_variant_gene_links(
links: Iterable[Mapping[str, Any] | VariantGeneLink],
) -> dict[str, Any]
Project one variant's links into the contract's columns; an empty links gives {} (no row).
Each link is a link_genes record, a mapping with the LINK_GENE_FIELDS as link_gene_record returns it and
a Parquet link dataset stores it, or a VariantGeneLink, which is converted with link_gene_record; every
reader projects through this one function. Links are ordered strongest first, honoring each score's
score_direction: a link whose score is missing or whose direction is unknown cannot be ranked and sorts
last. Ties, and links of different directions in one relation, resolve by record_id. The link_top_*
columns describe the first link when it can be ranked.
decode_link_cursor
¶
decode_link_cursor(
query: VariantGeneLinkQuery,
) -> str | None
Decode and validate query.cursor, returning its source record id.
encode_link_cursor
¶
encode_link_cursor(
query: VariantGeneLinkQuery, last_record_id: str
) -> str
Encode an opaque cursor bound to the query's locus and filters.
get_variant_gene_link_source_registry
¶
get_variant_gene_link_source_registry() -> (
PluginRegistry[type[VariantGeneLinkSource]]
)
Return the registry for altar.variant_gene_link_sources adapters.
normalize_chromosome
¶
Return a canonical chromosome label without a chr prefix.