Skip to content

Add an annotation source

An annotation source adds precomputed per-variant values, such as a score your lab computed, to Altar's output. It runs no model: it looks values up. The steps below build one over your own Parquet file. A link source, which returns several variant-to-gene relations per variant, is covered in Variant-to-gene links.

When to choose an annotation source

Write an annotation source when the values exist before the analysis starts, in a file, a database or a service; Model or source? says when to write a model binding instead. Each variant gets zero or one record, and a record may hold scalar, repeated or structured fields. If one variant links to several genes, each with its own context and provenance, write a link source.

A source you only use from your own Python code needs no package or entry point: pass the instance to altar.run.run or to a store's materialize(). Serve annotations from your own tables covers configuring the generic BigQuery and SQLite sources over tables you already have.

The interface

Subclass AnnotationSource, imported from altar.sources. It is Stable.

Member Required? What it is
name required The registry name, and the type a run spec uses.
annotation_columns() required The columns annotate() may return.
annotate(variant_ids) required Returns {variant_id: {column: value}} for the variants the source has a record for.
identity() default None; declare it The contract, data release and genome build the source serves. A source without one serves hg38 only.
supported_genome_builds default None (any assembly) Every assembly the class can serve.
from_options(**options) default: raises CapabilityNotSupported Builds the source from a run spec's plain options. Without it, a spec cannot name the source.
close() default: does nothing Releases what the source opened.
prioritize_predicate() default None A rule over the source's own columns that flags a variant.
add_annotations(rows), get_unannotated_variants(variant_ids) default: raise CapabilityNotSupported Only for a source that Altar also writes to.
annotation_contract() provisional; default None The contract every instance implements, so altar sources lists the columns without building the source.
option_descriptions provisional; default {} What each option of from_options means, by name, for altar sources.
published_dataset() provisional; default None The copy of the data its publisher hosts; see Publish a dataset from a binding.
vcf_info_fields() provisional; not on the base class A classmethod the augmented-VCF writer looks for; see Write VCF INFO fields.

A provisional member is expected to change; see Provisional names.

1. Write the source

You need Altar installed (pip install altar) and a Parquet file with one row per variant. The file needs a chromosome, position, reference allele and alternate allele column, plus the values you want to add. The examples use these columns:

chrom pos ref alt score
chr1 12345 A G 0.91
chr2 500 C T 0.05

To follow along without your own data, this writes that file and a two-variant cohort to the current directory:

import pyarrow as pa
import pyarrow.parquet as pq

pq.write_table(
    pa.table(
        {
            "chrom": ["chr1", "chr2"],
            "pos": [12345, 500],
            "ref": ["A", "C"],
            "alt": ["G", "T"],
            "score": [0.91, 0.05],
        }
    ),
    "lab_scores.parquet",
)
with open("cohort.tsv", "w") as cohort:
    cohort.write("chr1\t12345\tA\tG\nchr3\t100\tA\tC\n")

Put the source in a module of your own, such as lab_altar/source.py:

from __future__ import annotations

from collections.abc import Sequence
from typing import Any, Self

from altar.sources import (
    AllelicKeyColumns,
    AllelicRecordQuery,
    AnnotationColumn,
    AnnotationContract,
    AnnotationSource,
    AnnotationSourceIdentity,
    ParquetAllelicRecordBackend,
    allelic_key,
)

LAB_SCORE_ANNOTATIONS = AnnotationContract(
    source_id="org.example.annotation.lab-score",
    schema_version="1.0.0",
    columns=(AnnotationColumn("lab_score", "float", "Lab variant score"),),
)


class LabScoreSource(AnnotationSource):
    name = "lab_score"
    genome_build = "hg38"
    supported_genome_builds = frozenset({genome_build})

    def __init__(self, dataset: str, *, release: str) -> None:
        self.release = release
        self.backend = ParquetAllelicRecordBackend(
            dataset,
            key_columns=AllelicKeyColumns(
                chromosome="chrom", position="pos", reference_allele="ref", alternate_allele="alt"
            ),
        )

    @classmethod
    def from_options(cls, **options: Any) -> Self:
        return cls(**options)

    def annotation_columns(self) -> list[AnnotationColumn]:
        return list(LAB_SCORE_ANNOTATIONS.columns)

    def identity(self) -> AnnotationSourceIdentity:
        return LAB_SCORE_ANNOTATIONS.identity(release=self.release, genome_build=self.genome_build)

    def close(self) -> None:
        self.backend.close()

    async def annotate(self, variant_ids: Sequence[str]) -> dict[str, dict[str, object]]:
        keys = tuple(allelic_key(variant_id) for variant_id in dict.fromkeys(variant_ids))
        if not keys:
            return {}
        records = await self.backend.fetch(AllelicRecordQuery(keys=keys, fields=("score",)))
        annotations: dict[str, dict[str, object]] = {}
        for record in records:
            if record.variant_id in annotations:
                raise ValueError(f"lab_scores.parquet has more than one row for {record.variant_id}")
            annotations[record.variant_id] = {"lab_score": record.fields["score"]}
        return annotations

What each part does:

  • LAB_SCORE_ANNOTATIONS is the source's contract: a source_id, a semantic schema_version and the columns the source adds. Contracts and identity below gives the rules.
  • name is the source's registry name, and the name a run spec uses in type = "lab_score".
  • annotation_columns() declares the columns annotate() may return. Declare repeated and structured columns as such (repeated=True, dtype="struct"), even if a backend stores them as JSON.
  • identity() names the contract, the data release and the assembly the source serves. Change release whenever the file's values could change, such as a new upstream download: Altar keys reuse of annotated results by it.
  • supported_genome_builds lists every assembly the class can serve; identity() names the one this instance serves. See Assemblies below.
  • annotate() turns each canonical variant ID into the key the backend looks up with allelic_key, fetches the score field, and returns {variant_id: {column: value}}. Leave out a variant with no row: Altar fills it in as missing. Return an observed zero as zero.
  • from_options() builds the source from the plain values in a run spec. Without it, the source can still be used from Python, but a spec cannot name it.
  • close() closes the backend the constructor opened. See Close what the source opens.

ParquetAllelicRecordBackend only retrieves records; the source owns what they mean. When a variant can match several rows, such as one per transcript, fetch returns all of them and annotate() aggregates them. This example has one row per variant, so it rejects a duplicate rather than silently keep one. A backend never returns a variant you did not request.

allelic_key raises VariantIdentityError, a ValueError, for an ID that is not canonical chr<chromosome>:<position>:<ref>:<alt>. Altar hands annotate() only canonical IDs. If your file writes chromosomes without the chr prefix, rewrite the key's chromosome and keep its variant_id: dataclasses.replace(key, chromosome=key.chromosome.removeprefix("chr")). allelic_key is provisional.

Contracts and identity

An AnnotationContract is a published, versioned set of columns:

  • source_id is a reverse-domain name of at least three parts, in lowercase letters, digits, dots and hyphens, such as org.example.annotation.lab-score.
  • schema_version is a semantic version. Two versions with the same major version have compatible columns. Adding a column is a minor version. Removing or redefining one is a major version.
  • columns are the AnnotationColumn values every implementation returns.

contract.identity(release=..., genome_build=...) returns an AnnotationSourceIdentity: the contract, the data release and the assembly of the variant IDs the source answers for. identity_hash is a deterministic hash of those four fields. Altar records the identity in a run's provenance and keys reuse of stored annotations by it.

Several sources can implement one contract, such as a shipped binding and a warehouse table that holds the same columns. A model whose prioritization rule reads your columns names the contract by source_id, and Altar resolves it against the configured sources; see Annotation dependencies. Publish the contract as a module constant so others can import it, as the shipped bindings do: ALPHAMISSENSE_ANNOTATIONS in altar_alphamissense, SPLICEAI_ANNOTATIONS in altar_spliceai and GPNSTAR_ANNOTATIONS in altar_gpnstar.

Assemblies

A source serves variants on one assembly, which identity() names. altar run checks each source's genome_build against [run] genome_build before it reads the cohort, so an hg19 cohort is never annotated with hg38 values. The UCSC and GRC spellings (hg38 and GRCh38) name the same assembly.

Declare an identity on every source. A source with no identity() is taken to serve hg38 only: it is joined to hg38 variants and dropped for every other assembly.

supported_genome_builds tells Altar, when it assembles results, which assemblies a class could ever serve. When a model needs a contract on an assembly no source class supports, such as region annotations for mouse variants, its columns read as missing with an AnnotationUnavailableWarning instead of failing the run. Leave the attribute at its default, None, only when the class can serve any assembly. See Variant and training assemblies.

2. Use it from Python

Run this from the directory that holds lab_altar/ and the data. fetch_annotations runs one or more sources over a list of variant IDs:

import asyncio

from altar.sources import fetch_annotations
from lab_altar.source import LabScoreSource

source = LabScoreSource("lab_scores.parquet", release="2026-10")
annotations = asyncio.run(fetch_annotations([source], ["chr1:12345:A:G", "chr3:100:A:C"]))
print(annotations)
{'chr1:12345:A:G': {'lab_score': 0.91}, 'chr3:100:A:C': {}}

To annotate a whole cohort and export a table, pass the source to altar.run.run as annotation_sources. This spec, annotate.toml, reads the cohort and names no sources of its own:

[run]
job_id = "lab-scores"
genome_build = "hg38"

[variants]
path = "cohort.tsv"

[export]
format = "parquet"
from altar.run import run

summary = asyncio.run(run("annotate.toml", annotation_sources=[source]))
print(summary.export_uri)

The export holds a row per variant with a lab_score column. The run spec reference describes the other sections.

3. Register the entry point

altar run finds a source by its registry name, so the source has to be installed as a package that declares an entry point in the altar.annotation_sources group. A minimal pyproject.toml next to the lab_altar/ directory:

[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"

[project]
name = "lab-altar"
version = "0.1.0"
dependencies = ["altar"]

[project.entry-points."altar.annotation_sources"]
lab_score = "lab_altar.source:LabScoreSource"

The key, lab_score, must equal the class's name. Add an empty lab_altar/__init__.py, install the package with pip install -e ., and check that Altar sees it with altar sources. A spec then declares the source with type set to that name, and options holds the keyword arguments from_options receives:

[sources.lab]
type = "lab_score"
options = { dataset = "lab_scores.parquet", release = "2026-10" }

Add that section to annotate.toml and run altar run annotate.toml. Relative paths in options are resolved against the directory you run from. Options must be plain values: strings, numbers, booleans and paths. A source whose constructor needs a client or other live object leaves from_options out, and is passed in code as in step 2. A source that reads object storage or a warehouse can take a named connection in from_options instead of credentials; see Read through a named connection.

The shipped bindings name each option as a keyword-only parameter of from_options, and altar sources <name> lists those parameters as the source's options. They collect any other keyword in **unknown and raise ValueError(unknown_options_message("AlphaMissense", cls.from_options, unknown)), so a misspelled option names the valid ones. owned_dataset_storage_options returns the storage options that read a dataset through the named connection. Both helpers are in altar.sources, and both are provisional.

4. Test it with the conformance suite

altar.testing.AnnotationSourceContract is a pytest suite every source should pass. Subclass it in a test file, such as tests/test_lab_score.py, and provide the source, IDs it has rows for, and its contract:

import pytest

from altar.sources import AnnotationContract
from altar.testing import AnnotationSourceContract
from lab_altar.source import LAB_SCORE_ANNOTATIONS, LabScoreSource


class TestLabScoreSource(AnnotationSourceContract):
    @pytest.fixture
    def source(self) -> LabScoreSource:
        return LabScoreSource("lab_scores.parquet", release="test")

    @pytest.fixture
    def known_variant_ids(self) -> list[str]:
        return ["chr1:12345:A:G"]

    @pytest.fixture
    def contract(self) -> AnnotationContract:
        return LAB_SCORE_ANNOTATIONS

Run pytest from the directory that holds lab_scores.parquet, since the fixture names it by a relative path. The suite's async tests need pip install pytest-asyncio and this in pyproject.toml:

[tool.pytest.ini_options]
asyncio_mode = "auto"

The suite checks that the columns are well formed, that annotate() is keyed by the requested IDs and leaves out unknown ones, that the identity is valid and names a build in supported_genome_builds, and that the source implements the contract. Add your own tests for missing rows, zero values, null values and duplicate rows.

A link source returns variant-to-gene relations: for one variant, any number of genes or regulatory elements, each with its own biosample, score, distance and provenance. Subclass VariantGeneLinkSource from altar.sources. It is Stable. Set name, binding_id and binding_version, and implement query_links(query):

page = await source.query_links(query)
for link in page.links:
    use(link)

A VariantGeneLinkQuery names a locus, with its genome build, and optional filters such as genes, biosamples, methods, releases and a score range. query_links returns one bounded VariantGeneLinkPage of VariantGeneLink values and a next_cursor. A valid query with no links returns an empty page with no cursor. A network, storage or data failure raises VariantGeneLinkSourceError, never an empty page. encode_link_cursor and decode_link_cursor bind a cursor to its query, and the inherited stream_links walks every page.

Most link sources store their data first. Normalize each upstream row into a VariantGeneEvidenceRecord and write the records into a VariantGeneLinkStore. A store holds each release as a generation: write_records stages batches, and publish_generation makes the whole generation visible at once. InMemoryVariantGeneLinkStore and SqliteVariantGeneLinkStore ship with Altar. StoredVariantGeneLinkSource(store, dataset, name=..., binding_id=..., binding_version=...) then answers queries from the store, and the ENCODE-rE2G and scE2G atlas binding subclasses it. A record keeps a stable evidence interval. The source attaches the caller's query locus only when it returns a VariantGeneLink.

VariantGeneLinkAnnotationSource projects links into per-variant annotation columns, so a run spec can export them. It reads a link store, a Parquet link dataset (from_dataset, provisional) or that dataset loaded into BigQuery (from_bigquery_table, provisional). Open Targets E2G reads its published dataset this way.

Register a link source under the altar.variant_gene_link_sources entry-point group, with its name as the key. Test it with VariantGeneLinkSourceContract, a store with VariantGeneLinkStoreContract, and a projection with VariantGeneLinkAnnotationSourceContract.

Going further

Prioritize variants

Return None from prioritize_predicate(), the default, for passive evidence. If the source has a documented policy, return a predicate from altar.predicates, and expose any threshold as a setting:

from altar.predicates import Col, Ge, Lit


def prioritize_predicate(self):
    return Ge(Col("lab_score"), Lit(self.threshold))

Close what the source opens

AnnotationSource.close() releases what the source opened, and a with block calls it on exit. The default does nothing. A source that opens a connection, a client or a backend overrides it, and a second call must do nothing.

A source closes only what it opened. A backend or client passed to its constructor belongs to the caller, who closes it. LabScoreSource builds its own ParquetAllelicRecordBackend, so it closes it.

altar run closes each source it builds from a [sources.<name>] section once, when the run ends: after the export, after a failed check or model, and when the run is cancelled. A source passed to altar.run.run as annotation_sources stays open, so you can use it again and close it yourself:

with LabScoreSource("lab_scores.parquet", release="2026-10") as source:
    asyncio.run(run("annotate.toml", annotation_sources=[source]))

A test fixture over a source that holds a connection yields it and closes it after the test.

Read the dataset from elsewhere

ParquetAllelicRecordBackend reads one Parquet file or a directory of them. With a directory partitioned as chrom=chr1/, a lookup opens only the files of the requested chromosomes, and reads only the row groups whose position statistics can hold a requested key. Sort rows by position within a chromosome to keep those statistics tight.

dataset can be a local path or a URI: gs://, s3://, az://, https://, hf://, or memory://. A remote URI is read through fsspec, so install the provider's extra (altar[gcs], altar[s3], altar[azure], altar[http], or altar[hf]). storage_options is passed to that filesystem unchanged. Use it only for non-secret settings such as {"token": "anon"} or a requester-pays project, and leave credentials ambient. A remote dataset is listed once per backend instance and treated as an unchanging release. An HTTP(S) server that returns no directory index can serve a single .parquet file; a partitioned directory needs an object store or hf://.

The source takes any AllelicRecordBackend, not only Parquet. Altar also ships SqliteAllelicRecordBackend(database, table=..., key_columns=...) and BigQueryAllelicRecordBackend(client, table_id, key_columns=..., filters=...), and all three pass the same conformance suite (altar.testing.AllelicRecordBackendContract). Accept a backend in the constructor, rather than writing a Parquet, SQLite and BigQuery variant of the source: one scientific source composes with any storage adapter. The AlphaMissense, SpliceAI, ClinVar and population-frequency bindings follow this pattern.

Read through a named connection

A source that reads object storage or a warehouse takes a connection keyword in from_options. A run spec's [sources.<name>] section names the connection with connection = "<name>", and the run driver passes the live object. The source asks it for what it needs:

from typing import Self

from altar.connections import Connection
from altar.sources import AnnotationSource


class ConservationTable(AnnotationSource):
    name = "conservation"

    @classmethod
    def from_options(cls, *, dataset: str, connection: Connection | None = None) -> Self:
        storage_options = {} if connection is None else connection.storage_options(dataset)
        return cls(dataset, storage_options=storage_options)

storage_options(uri) returns the fsspec keyword arguments for a URI. bigquery_client() returns a new BigQuery client, which the source then owns and closes. A connection type that cannot reach what you ask for raises CapabilityNotSupported. A source that also accepts storage_options refuses them together with a connection, so the two never disagree.

Keep the connection out of identity(). The identity names the data release, and the connection only says how to fetch it. Connection is provisional; Add connections and published datasets covers writing a connection type and publishing a source's data.

Join a source in BigQuery

BigQueryScoreStore joins an annotation source into the SQL queries that assemble and export results. A source opts in by subclassing SqlAnnotationSource and implementing sql_join(ctx), which returns the JoinSpec the store adds to its query, or None to be read through annotate() instead. Test it with SqlAnnotationSourceContract.

You rarely write one. For a table that only needs columns selected, renamed or aggregated, configure a BigQueryAnnotationSource with a relation= subquery. For a sparse source whose binding computes its annotations in Python, open staged_annotation_source with the job's variants, and pass the source it yields to the store inside the async with block. Pass the BigQueryAllelicRecordBackend as coverage= so only variants with records are annotated. Run at warehouse scale on BigQuery describes when each fits.

Describe the source to altar sources

altar sources <name> describes a source without building it, so it needs no data, credentials or network. It reads the first paragraph of the class's docstring, the keyword parameters of from_options and their Literal values, and the class's contract. The provisional class attribute option_descriptions maps an option's name to what it means, a short phrase in lower case:

option_descriptions: ClassVar[Mapping[str, str]] = {
    "dataset": "the copy `altar-lab-build` wrote, as a path or URI",
}

altar sources shows each phrase beside its option, and the integration page's options table shows it as the option's meaning. Every key must name a keyword parameter of from_options, which AnnotationSourceContract checks. Return the contract from the provisional classmethod annotation_contract(), and the command lists its columns. A class that returns one promises that every instance's identity() names that contract, and AnnotationSourceContract checks it.

Write VCF INFO fields

altar.variants.augment_run_export writes an augmented VCF from an altar run export. For each source in the export it looks the source's class up by name and calls its classmethod vcf_info_fields(), if it has one. The method returns a tuple of AnnotationInfoField values, each naming an INFO ID, the source columns it reads, its VCF type and its description. The base class does not define the method; the writer looks for it by name. The hook and AnnotationInfoField are provisional. Declare annotation fields describes the fields.

A source Altar writes to

A source that caches annotations a pipeline computes implements add_annotations(rows) and get_unannotated_variants(variant_ids). A source with an identity counts only rows written under that identity, so a new release reports every variant as unannotated. Test it with WritableAnnotationSourceContract as well as AnnotationSourceContract. The generic BigQueryAnnotationSource and SqliteAnnotationSource already do this; Serve annotations from your own tables shows how to run them.

Ready when

  • The source declares an AnnotationContract with a namespaced source_id, and identity() returns the contract's identity with a release that changes whenever the values could.
  • supported_genome_builds lists every assembly the class can serve.
  • annotate() leaves out variants without a record, returns an observed zero as zero, and handles duplicate records on purpose.
  • from_options takes only plain values, and refuses unknown options with a message that names the valid ones.
  • The entry point's key equals name, and altar sources lists the source after pip install -e ..
  • AnnotationSourceContract passes, with the contract fixture set.
  • The source closes only what it opened, and a second close() does nothing.
  • The source's page states its columns, missing-value meaning, release, genome builds, citation and terms. Write an integration page gives the template and its generator.

Next