Add an annotation source¶
An annotation source adds precomputed per-variant values, such as a score your lab computed, to Altar's output. It runs no model: it looks values up. The steps below build one over your own Parquet file. A link source, which returns several variant-to-gene relations per variant, is covered in Variant-to-gene links.
When to choose an annotation source¶
Write an annotation source when the values exist before the analysis starts, in a file, a database or a service; Model or source? says when to write a model binding instead. Each variant gets zero or one record, and a record may hold scalar, repeated or structured fields. If one variant links to several genes, each with its own context and provenance, write a link source.
A source you only use from your own Python code needs no package or entry point: pass the instance to altar.run.run or
to a store's materialize(). Serve annotations from your own tables covers
configuring the generic BigQuery and SQLite sources over tables you already have.
The interface¶
Subclass AnnotationSource, imported from altar.sources. It is Stable.
| Member | Required? | What it is |
|---|---|---|
name |
required | The registry name, and the type a run spec uses. |
annotation_columns() |
required | The columns annotate() may return. |
annotate(variant_ids) |
required | Returns {variant_id: {column: value}} for the variants the source has a record for. |
identity() |
default None; declare it |
The contract, data release and genome build the source serves. A source without one serves hg38 only. |
supported_genome_builds |
default None (any assembly) |
Every assembly the class can serve. |
from_options(**options) |
default: raises CapabilityNotSupported |
Builds the source from a run spec's plain options. Without it, a spec cannot name the source. |
close() |
default: does nothing | Releases what the source opened. |
prioritize_predicate() |
default None |
A rule over the source's own columns that flags a variant. |
add_annotations(rows), get_unannotated_variants(variant_ids) |
default: raise CapabilityNotSupported |
Only for a source that Altar also writes to. |
annotation_contract() |
provisional; default None |
The contract every instance implements, so altar sources lists the columns without building the source. |
option_descriptions |
provisional; default {} |
What each option of from_options means, by name, for altar sources. |
published_dataset() |
provisional; default None |
The copy of the data its publisher hosts; see Publish a dataset from a binding. |
vcf_info_fields() |
provisional; not on the base class | A classmethod the augmented-VCF writer looks for; see Write VCF INFO fields. |
A provisional member is expected to change; see Provisional names.
1. Write the source¶
You need Altar installed (pip install altar) and a Parquet file with one row per variant. The file needs a
chromosome, position, reference allele and alternate allele column, plus the values you want to add. The examples
use these columns:
| chrom | pos | ref | alt | score |
|---|---|---|---|---|
| chr1 | 12345 | A | G | 0.91 |
| chr2 | 500 | C | T | 0.05 |
To follow along without your own data, this writes that file and a two-variant cohort to the current directory:
import pyarrow as pa
import pyarrow.parquet as pq
pq.write_table(
pa.table(
{
"chrom": ["chr1", "chr2"],
"pos": [12345, 500],
"ref": ["A", "C"],
"alt": ["G", "T"],
"score": [0.91, 0.05],
}
),
"lab_scores.parquet",
)
with open("cohort.tsv", "w") as cohort:
cohort.write("chr1\t12345\tA\tG\nchr3\t100\tA\tC\n")
Put the source in a module of your own, such as lab_altar/source.py:
from __future__ import annotations
from collections.abc import Sequence
from typing import Any, Self
from altar.sources import (
AllelicKeyColumns,
AllelicRecordQuery,
AnnotationColumn,
AnnotationContract,
AnnotationSource,
AnnotationSourceIdentity,
ParquetAllelicRecordBackend,
allelic_key,
)
LAB_SCORE_ANNOTATIONS = AnnotationContract(
source_id="org.example.annotation.lab-score",
schema_version="1.0.0",
columns=(AnnotationColumn("lab_score", "float", "Lab variant score"),),
)
class LabScoreSource(AnnotationSource):
name = "lab_score"
genome_build = "hg38"
supported_genome_builds = frozenset({genome_build})
def __init__(self, dataset: str, *, release: str) -> None:
self.release = release
self.backend = ParquetAllelicRecordBackend(
dataset,
key_columns=AllelicKeyColumns(
chromosome="chrom", position="pos", reference_allele="ref", alternate_allele="alt"
),
)
@classmethod
def from_options(cls, **options: Any) -> Self:
return cls(**options)
def annotation_columns(self) -> list[AnnotationColumn]:
return list(LAB_SCORE_ANNOTATIONS.columns)
def identity(self) -> AnnotationSourceIdentity:
return LAB_SCORE_ANNOTATIONS.identity(release=self.release, genome_build=self.genome_build)
def close(self) -> None:
self.backend.close()
async def annotate(self, variant_ids: Sequence[str]) -> dict[str, dict[str, object]]:
keys = tuple(allelic_key(variant_id) for variant_id in dict.fromkeys(variant_ids))
if not keys:
return {}
records = await self.backend.fetch(AllelicRecordQuery(keys=keys, fields=("score",)))
annotations: dict[str, dict[str, object]] = {}
for record in records:
if record.variant_id in annotations:
raise ValueError(f"lab_scores.parquet has more than one row for {record.variant_id}")
annotations[record.variant_id] = {"lab_score": record.fields["score"]}
return annotations
What each part does:
LAB_SCORE_ANNOTATIONSis the source's contract: asource_id, a semanticschema_versionand the columns the source adds. Contracts and identity below gives the rules.nameis the source's registry name, and the name a run spec uses intype = "lab_score".annotation_columns()declares the columnsannotate()may return. Declare repeated and structured columns as such (repeated=True,dtype="struct"), even if a backend stores them as JSON.identity()names the contract, the datareleaseand the assembly the source serves. Changereleasewhenever the file's values could change, such as a new upstream download: Altar keys reuse of annotated results by it.supported_genome_buildslists every assembly the class can serve;identity()names the one this instance serves. See Assemblies below.annotate()turns each canonical variant ID into the key the backend looks up withallelic_key, fetches thescorefield, and returns{variant_id: {column: value}}. Leave out a variant with no row: Altar fills it in as missing. Return an observed zero as zero.from_options()builds the source from the plain values in a run spec. Without it, the source can still be used from Python, but a spec cannot name it.close()closes the backend the constructor opened. See Close what the source opens.
ParquetAllelicRecordBackend only retrieves records; the source owns what they mean. When a variant can match
several rows, such as one per transcript, fetch returns all of them and annotate() aggregates them. This
example has one row per variant, so it rejects a duplicate rather than silently keep one. A backend never
returns a variant you did not request.
allelic_key raises VariantIdentityError, a ValueError, for an ID that is not canonical
chr<chromosome>:<position>:<ref>:<alt>. Altar hands annotate() only canonical IDs. If your file writes
chromosomes without the chr prefix, rewrite the key's chromosome and keep its variant_id:
dataclasses.replace(key, chromosome=key.chromosome.removeprefix("chr")). allelic_key is
provisional.
Contracts and identity¶
An AnnotationContract is a published, versioned set of columns:
source_idis a reverse-domain name of at least three parts, in lowercase letters, digits, dots and hyphens, such asorg.example.annotation.lab-score.schema_versionis a semantic version. Two versions with the same major version have compatible columns. Adding a column is a minor version. Removing or redefining one is a major version.columnsare theAnnotationColumnvalues every implementation returns.
contract.identity(release=..., genome_build=...) returns an AnnotationSourceIdentity: the contract, the data
release and the assembly of the variant IDs the source answers for. identity_hash is a deterministic hash of
those four fields. Altar records the identity in a run's provenance and keys reuse of stored annotations by it.
Several sources can implement one contract, such as a shipped binding and a warehouse table that holds the same
columns. A model whose prioritization rule reads your columns names the contract by source_id, and Altar resolves
it against the configured sources; see
Annotation dependencies. Publish the contract as a module
constant so others can import it, as the shipped bindings do: ALPHAMISSENSE_ANNOTATIONS in
altar_alphamissense, SPLICEAI_ANNOTATIONS in altar_spliceai and GPNSTAR_ANNOTATIONS in altar_gpnstar.
Assemblies¶
A source serves variants on one assembly, which identity() names. altar run checks each source's
genome_build against [run] genome_build before it reads the cohort, so an hg19 cohort is never annotated
with hg38 values. The UCSC and GRC spellings (hg38 and GRCh38) name the same assembly.
Declare an identity on every source. A source with no identity() is taken to serve hg38 only: it is joined
to hg38 variants and dropped for every other assembly.
supported_genome_builds tells Altar, when it assembles results, which assemblies a class could ever serve. When a
model needs a contract on an assembly no source class supports, such as region annotations for mouse variants, its
columns read as missing with an AnnotationUnavailableWarning instead of failing the run. Leave the attribute at its
default, None, only when the class can serve any assembly. See
Variant and training assemblies.
2. Use it from Python¶
Run this from the directory that holds lab_altar/ and the data. fetch_annotations runs one or more sources
over a list of variant IDs:
import asyncio
from altar.sources import fetch_annotations
from lab_altar.source import LabScoreSource
source = LabScoreSource("lab_scores.parquet", release="2026-10")
annotations = asyncio.run(fetch_annotations([source], ["chr1:12345:A:G", "chr3:100:A:C"]))
print(annotations)
To annotate a whole cohort and export a table, pass the source to altar.run.run as annotation_sources. This
spec, annotate.toml, reads the cohort and names no sources of its own:
[run]
job_id = "lab-scores"
genome_build = "hg38"
[variants]
path = "cohort.tsv"
[export]
format = "parquet"
from altar.run import run
summary = asyncio.run(run("annotate.toml", annotation_sources=[source]))
print(summary.export_uri)
The export holds a row per variant with a lab_score column. The run spec reference
describes the other sections.
3. Register the entry point¶
altar run finds a source by its registry name, so the source has to be installed as a package that declares an
entry point in the altar.annotation_sources group. A minimal pyproject.toml next to the lab_altar/ directory:
[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"
[project]
name = "lab-altar"
version = "0.1.0"
dependencies = ["altar"]
[project.entry-points."altar.annotation_sources"]
lab_score = "lab_altar.source:LabScoreSource"
The key, lab_score, must equal the class's name. Add an empty lab_altar/__init__.py, install the package
with pip install -e ., and check that Altar sees it with altar sources. A spec then declares the source with
type set to that name, and options holds the keyword arguments from_options receives:
Add that section to annotate.toml and run altar run annotate.toml. Relative paths in options are resolved
against the directory you run from. Options must be plain values: strings, numbers, booleans and paths. A source
whose constructor needs a client or other live object leaves from_options out, and is passed in code as in
step 2. A source that reads object storage or a warehouse can take a named connection in from_options instead of
credentials; see Read through a named connection.
The shipped bindings name each option as a keyword-only parameter of from_options, and altar sources
<name> lists those parameters as the source's options. They collect any other keyword in **unknown and raise
ValueError(unknown_options_message("AlphaMissense", cls.from_options, unknown)), so a misspelled option names
the valid ones. owned_dataset_storage_options returns the storage options that read a dataset through the
named connection. Both helpers are in altar.sources, and both are provisional.
4. Test it with the conformance suite¶
altar.testing.AnnotationSourceContract is a pytest suite every source should pass. Subclass it in a test
file, such as tests/test_lab_score.py, and provide the source, IDs it has rows for, and its contract:
import pytest
from altar.sources import AnnotationContract
from altar.testing import AnnotationSourceContract
from lab_altar.source import LAB_SCORE_ANNOTATIONS, LabScoreSource
class TestLabScoreSource(AnnotationSourceContract):
@pytest.fixture
def source(self) -> LabScoreSource:
return LabScoreSource("lab_scores.parquet", release="test")
@pytest.fixture
def known_variant_ids(self) -> list[str]:
return ["chr1:12345:A:G"]
@pytest.fixture
def contract(self) -> AnnotationContract:
return LAB_SCORE_ANNOTATIONS
Run pytest from the directory that holds lab_scores.parquet, since the fixture names it by a relative path.
The suite's async tests need pip install pytest-asyncio and this in pyproject.toml:
The suite checks that the columns are well formed, that annotate() is keyed by the requested IDs and leaves out
unknown ones, that the identity is valid and names a build in supported_genome_builds, and that the source
implements the contract. Add your own tests for missing rows, zero values, null values and duplicate rows.
Variant-to-gene links¶
A link source returns variant-to-gene relations: for one variant, any number of genes or regulatory elements,
each with its own biosample, score, distance and provenance. Subclass VariantGeneLinkSource from altar.sources.
It is Stable. Set name, binding_id and binding_version, and implement query_links(query):
A VariantGeneLinkQuery names a locus, with its genome build, and optional filters such as genes, biosamples,
methods, releases and a score range. query_links returns one bounded VariantGeneLinkPage of VariantGeneLink
values and a next_cursor. A valid query with no links returns an empty page with no cursor. A network, storage or
data failure raises VariantGeneLinkSourceError, never an empty page. encode_link_cursor and
decode_link_cursor bind a cursor to its query, and the inherited stream_links walks every page.
Most link sources store their data first. Normalize each upstream row into a VariantGeneEvidenceRecord and write
the records into a VariantGeneLinkStore. A store holds each release as a generation: write_records stages
batches, and publish_generation makes the whole generation visible at once. InMemoryVariantGeneLinkStore and
SqliteVariantGeneLinkStore ship with Altar. StoredVariantGeneLinkSource(store, dataset, name=..., binding_id=...,
binding_version=...) then answers queries from the store, and the ENCODE-rE2G and scE2G atlas binding subclasses
it. A record keeps a stable evidence interval. The source attaches the caller's query locus only when it returns a
VariantGeneLink.
VariantGeneLinkAnnotationSource projects links into per-variant annotation columns, so a run spec can export
them. It reads a link store, a Parquet link dataset (from_dataset, provisional) or that dataset loaded into
BigQuery (from_bigquery_table, provisional). Open Targets E2G reads its
published dataset this way.
Register a link source under the altar.variant_gene_link_sources entry-point group, with its name as the key.
Test it with VariantGeneLinkSourceContract, a store with VariantGeneLinkStoreContract, and a projection with
VariantGeneLinkAnnotationSourceContract.
Going further¶
Prioritize variants¶
Return None from prioritize_predicate(), the default, for passive evidence. If the source has a documented
policy, return a predicate from altar.predicates, and expose any threshold as a setting:
from altar.predicates import Col, Ge, Lit
def prioritize_predicate(self):
return Ge(Col("lab_score"), Lit(self.threshold))
Close what the source opens¶
AnnotationSource.close() releases what the source opened, and a with block calls it on exit. The default does
nothing. A source that opens a connection, a client or a backend overrides it, and a second call must do nothing.
A source closes only what it opened. A backend or client passed to its constructor belongs to the caller, who
closes it. LabScoreSource builds its own ParquetAllelicRecordBackend, so it closes it.
altar run closes each source it builds from a [sources.<name>] section once, when the run ends: after the
export, after a failed check or model, and when the run is cancelled. A source passed to altar.run.run as
annotation_sources stays open, so you can use it again and close it yourself:
with LabScoreSource("lab_scores.parquet", release="2026-10") as source:
asyncio.run(run("annotate.toml", annotation_sources=[source]))
A test fixture over a source that holds a connection yields it and closes it after the test.
Read the dataset from elsewhere¶
ParquetAllelicRecordBackend reads one Parquet file or a directory of them. With a directory partitioned as
chrom=chr1/, a lookup opens only the files of the requested chromosomes, and reads only the row groups whose
position statistics can hold a requested key. Sort rows by position within a chromosome to keep those statistics
tight.
dataset can be a local path or a URI: gs://, s3://, az://, https://, hf://, or memory://. A remote
URI is read through fsspec, so install the provider's extra (altar[gcs], altar[s3], altar[azure],
altar[http], or altar[hf]). storage_options is passed to that filesystem unchanged. Use it only for
non-secret settings such as {"token": "anon"} or a requester-pays project, and leave credentials ambient. A
remote dataset is listed once per backend instance and treated as an unchanging release. An HTTP(S) server that
returns no directory index can serve a single .parquet file; a partitioned directory needs an object store or
hf://.
The source takes any AllelicRecordBackend, not only Parquet. Altar also ships
SqliteAllelicRecordBackend(database, table=..., key_columns=...) and
BigQueryAllelicRecordBackend(client, table_id, key_columns=..., filters=...), and all three pass the same
conformance suite (altar.testing.AllelicRecordBackendContract). Accept a backend in the constructor, rather than
writing a Parquet, SQLite and BigQuery variant of the source: one scientific source composes with any storage
adapter. The AlphaMissense, SpliceAI, ClinVar and population-frequency bindings follow this pattern.
Read through a named connection¶
A source that reads object storage or a warehouse takes a connection keyword in from_options. A run spec's
[sources.<name>] section names the connection with connection = "<name>", and the run driver passes the live
object. The source asks it for what it needs:
from typing import Self
from altar.connections import Connection
from altar.sources import AnnotationSource
class ConservationTable(AnnotationSource):
name = "conservation"
@classmethod
def from_options(cls, *, dataset: str, connection: Connection | None = None) -> Self:
storage_options = {} if connection is None else connection.storage_options(dataset)
return cls(dataset, storage_options=storage_options)
storage_options(uri) returns the fsspec keyword arguments for a URI. bigquery_client() returns a new BigQuery
client, which the source then owns and closes. A connection type that cannot reach what you ask for raises
CapabilityNotSupported. A source that also accepts storage_options refuses them together with a connection,
so the two never disagree.
Keep the connection out of identity(). The identity names the data release, and the connection only says how to
fetch it. Connection is provisional; Add connections and published datasets
covers writing a connection type and publishing a source's data.
Join a source in BigQuery¶
BigQueryScoreStore joins an annotation source into the SQL queries that assemble and export results. A source opts in
by subclassing SqlAnnotationSource and implementing sql_join(ctx), which returns the JoinSpec the store adds to
its query, or None to be read through annotate() instead. Test it with SqlAnnotationSourceContract.
You rarely write one. For a table that only needs columns selected, renamed or aggregated, configure a
BigQueryAnnotationSource with a relation= subquery. For a sparse source whose binding computes its annotations
in Python, open staged_annotation_source with the job's variants, and pass the source it yields to the store
inside the async with block. Pass the BigQueryAllelicRecordBackend as coverage= so only variants with records
are annotated. Run at warehouse scale on BigQuery describes when each fits.
Describe the source to altar sources¶
altar sources <name> describes a source without building it, so it needs no data, credentials or network. It
reads the first paragraph of the class's docstring, the keyword parameters of from_options and their Literal
values, and the class's contract. The provisional class attribute option_descriptions maps an option's name to
what it means, a short phrase in lower case:
option_descriptions: ClassVar[Mapping[str, str]] = {
"dataset": "the copy `altar-lab-build` wrote, as a path or URI",
}
altar sources shows each phrase beside its option, and the integration page's options table shows it as the
option's meaning. Every key must name a keyword parameter of from_options, which AnnotationSourceContract
checks. Return the contract from the provisional classmethod annotation_contract(),
and the command lists its columns. A class that returns one promises that every instance's identity() names that
contract, and AnnotationSourceContract checks it.
Write VCF INFO fields¶
altar.variants.augment_run_export writes an augmented VCF from an altar run export. For each source in the
export it looks the source's class up by name and calls its classmethod vcf_info_fields(), if it has one. The
method returns a tuple of AnnotationInfoField values, each naming an INFO ID, the source columns it reads, its
VCF type and its description. The base class does not define the method; the writer looks for it by name. The hook
and AnnotationInfoField are provisional. Declare annotation fields describes the fields.
A source Altar writes to¶
A source that caches annotations a pipeline computes implements add_annotations(rows) and
get_unannotated_variants(variant_ids). A source with an identity counts only rows written under that identity, so a
new release reports every variant as unannotated. Test it with WritableAnnotationSourceContract as well as
AnnotationSourceContract. The generic BigQueryAnnotationSource and SqliteAnnotationSource already do this;
Serve annotations from your own tables
shows how to run them.
Ready when¶
- The source declares an
AnnotationContractwith a namespacedsource_id, andidentity()returns the contract's identity with areleasethat changes whenever the values could. supported_genome_buildslists every assembly the class can serve.annotate()leaves out variants without a record, returns an observed zero as zero, and handles duplicate records on purpose.from_optionstakes only plain values, and refuses unknown options with a message that names the valid ones.- The entry point's key equals
name, andaltar sourceslists the source afterpip install -e .. AnnotationSourceContractpasses, with thecontractfixture set.- The source closes only what it opened, and a second
close()does nothing. - The source's page states its columns, missing-value meaning, release, genome builds, citation and terms. Write an integration page gives the template and its generator.
Next¶
- Package and register a plugin covers entry-point groups and the plugin registry.
- Validate an implementation lists every conformance suite.