Skip to content

Serve annotations from your own tables

Your own application can serve annotations from tables it maintains and combine them with the annotation sources Altar ships, such as AlphaMissense. A source over your own table is an ordinary AnnotationSource that you pass to materialize or to altar.run.run. It needs no package, registry entry or entry point.

Altar checks, joins and prioritizes on what a source declares: its columns, from annotation_columns(), and its identity, from identity(). It never looks at the source's class or registry name. To write a source class, see Add an annotation source. To test one with the conformance suite, see Validate an implementation.

Everything here is in altar.sources and is Stable, except BigQueryAnnotationSource, which is Preview; see Run at warehouse scale on BigQuery.

Declare a contract and an identity

A table you serve implements an AnnotationContract: a published, versioned set of columns named by a source_id. contract.identity(release=..., genome_build=...) gives the AnnotationSourceIdentity a source declares: the contract, the data release, and the assembly of the variant IDs it answers for. Contracts and identity gives the naming and versioning rules. The examples on this page use this contract:

from altar.sources import AnnotationColumn, AnnotationContract

REGULATORY = AnnotationContract(
    source_id="org.example.annotation.regulatory",
    schema_version="1.0.0",
    columns=(
        AnnotationColumn("regulatory_class", "str", "Regulatory class"),
        AnnotationColumn("regulatory_score", "float", "Regulatory score"),
    ),
)
identity = REGULATORY.identity(release="2026-09", genome_build="hg38")

Change release whenever the values the table returns could change, such as after a new upstream download. identity.identity_hash cannot be turned back into the release; to find the hash of a release, build its identity and read identity_hash. Declare an identity on every source, because materialization joins a source only to variants on its assembly (Assemblies). To serve a shipped binding's columns from your own table, implement its contract, such as ALPHAMISSENSE_ANNOTATIONS from altar_alphamissense or altar.variants.REGION_ANNOTATION_CONTRACT.

Configure a generic source over your table

When a table already has one row per variant_id, configure BigQueryAnnotationSource or SqliteAnnotationSource instead of writing a class. Pass identity= to declare the contract the table implements:

from altar.sources import BigQueryAnnotationSource

regulatory = BigQueryAnnotationSource.connect(
    "my-project.annotations.regulatory_v2026_09",
    list(REGULATORY.columns),
    name="regulatory",
    identity=REGULATORY.identity(release="2026-09", genome_build="hg38"),
)

connect builds a BigQuery client from ambient credentials and needs the altar[bigquery] extra. To pass a client you built, call the constructor, BigQueryAnnotationSource(client, table_id, columns, ...). A relation= subquery selects, renames or aggregates the table's columns; see Fused sources.

On its own, identity= is metadata. It does not change how a source writes, deduplicates or reads rows. To make the identity decide which rows a table serves and counts, also pass identity_scoped=True; see Keep a writable annotation table current across releases.

identity.genome_build and genome_default are separate settings. identity.genome_build is the build label that dependency resolution compares with your models' runs, such as hg38. genome_default is how a writable table spells the build in its stored genome column. Altar's VCF ingest writes GRCh38 there. A table whose rows say GRCh38 keeps genome_default="GRCh38" and can still declare an hg38 identity. Neither setting fills in the other.

A source that reads a dataset in object storage or a warehouse can take a named connection; see Pass credentials through connections. The connection is not part of the identity, so one release read two ways keeps one identity.

Keep a writable annotation table current across releases

A writable annotation table is a table Altar writes annotations into and then reads back as a cache, such as the region labels your annotation jobs compute. Pass identity_scoped=True with identity= to keep each release's rows apart. Both generic sources take the flag:

  • A scoped BigQueryAnnotationSource stores the identity's hash on every row and uses only rows with that hash.
  • A scoped SqliteAnnotationSource records one identity per file and refuses to open the file under another.

In both, changing release makes every variant a cache miss, and your pipeline annotates them again. Without the flag, identity= is metadata on both.

A scoped BigQueryAnnotationSource works like this:

  • ensure_schema() adds a nullable _altar_annotation_identity STRING column. The name is altar.sources.ANNOTATION_IDENTITY_COLUMN.
  • add_annotations, add_annotations_frame and add_annotations_from_uri write the identity's identity_hash into that column on every row. A row that already carries a different hash raises AnnotationIdentityError.
  • get_unannotated_variants and export_unannotated count a variant as annotated only when it has a row with that hash.
  • annotate and materialization read only rows with that hash, so a table that holds several releases still joins one row per variant.
from altar.sources import BigQueryAnnotationSource
from altar.variants import REGION_ANNOTATION_CONTRACT

regions = BigQueryAnnotationSource.connect(
    "my-project.annotations.variant_regions",
    list(REGION_ANNOTATION_CONTRACT.columns),
    name="regions",
    genome_default="GRCh38",
    identity=REGION_ANNOTATION_CONTRACT.identity(release="gencode-v47-screen-v4", genome_build="hg38"),
    identity_scoped=True,
)


async def unannotated_variants():
    await regions.ensure_schema()
    return await regions.get_unannotated_variants([], variants_relation="`my-project.jobs.variants_123`")

The examples below use this regions source.

Every write stores the row's genome and its chr, pos, ref and alt beside the canonical variant_id, and get_unannotated_variants counts only rows of the source's genome_default build.

Scope every source object that reads or writes the table, with the same identity. A reader you configure separately from the writer needs identity_scoped=True too. Otherwise it reads every release's rows, and a variant can be served any release's values.

An unscoped BigQueryAnnotationSource checks for this. It reads its table's schema, a metadata request that runs no query and bills nothing. When the table has the _altar_annotation_identity column, the unscoped source:

  • refuses to write to it. add_annotations, add_annotations_frame and add_annotations_from_uri raise AnnotationIdentityError before writing or loading anything, because the table's scoped sources would neither serve nor count rows without the hash. A SQLite table refuses an unscoped write the same way.
  • warns before its first read or deduplication. Materialization, the BigQuery exports, annotate(), get_unannotated_variants and export_unannotated emit UnscopedAnnotationReadWarning and then run. An unscoped deduplication counts a variant annotated under any release as annotated. Each source object warns at most once.

If the schema lookup fails, for example for lack of permission, Altar logs it at debug level and runs unchecked. To make the warning fail a job, turn it into an error:

import warnings

from altar.sources import UnscopedAnnotationReadWarning

warnings.simplefilter("error", UnscopedAnnotationReadWarning)

Load annotation files from object storage

export_unannotated writes an annotation job's input. For each variant in your job relation that the table has no row for, it writes a headerless, tab-separated line of chr, pos, ref, alt, variant_id to object storage, and it returns how many lines it wrote. The first four columns are split from the variant_id, so the relation needs only a variant_id column. The region annotate command copies the fifth column into each JSONL row's variant_id, which the load below requires.

async def export_job_input():
    return await regions.export_unannotated(
        destination_uri="gs://my-bucket/run-7/unannotated-*.tsv",
        variants_relation="`my-project.jobs.variants_123`",
    )

export_unannotated and get_unannotated_variants never report a variant whose variant_id is not canonical, such as chr9:21980945:C:C (REF equal to ALT) or 9:100:C:T, because no write path can store one. export_unannotated logs a warning with how many it left out and an example. Delete them from the tables your job relation reads.

BigQuery's EXPORT DATA may write several files under the wildcard, and some may be empty. The annotate command reads one file, so each job runs these steps:

  1. Export to a fresh prefix for this job, such as gs://my-bucket/run-7/. The export replaces only the files it writes. It never deletes stale files an earlier export left under the same wildcard, and an export that finds nothing to annotate writes no file. Reusing a prefix can therefore annotate an earlier job's variants again.
  2. Annotate each unannotated-*.tsv file into its own JSONL, such as annotated-000000000000.jsonl. An empty file gives an empty JSONL. You can instead concatenate the files and annotate one.
  3. Load every JSONL of the job with one call to add_annotations_from_uri, passing one expected_identity. Every file's identity sidecar names the same identity when one annotate image produced them.
from altar.sources import load_annotation_identity


async def load_job_output(sidecar_bytes: bytes):
    # sidecar_bytes: gs://my-bucket/run-7/annotated.jsonl.identity.json, read with your own storage client.
    produced_under = load_annotation_identity(sidecar_bytes)
    result = await regions.add_annotations_from_uri("gs://my-bucket/run-7/*.jsonl", expected_identity=produced_under)
    print(result.added, result.skipped)

load_annotation_identity reads the identity sidecar the annotate command writes next to its JSONL. It takes the sidecar's text or bytes, its parsed object, or a local pathlib.Path, and checks the sidecar's format and that its identity_hash matches its identity. Altar does not read object storage here, so read the sidecar with your own client.

add_annotations_from_uri takes any URI, or list of URIs, a BigQuery load job accepts. It:

  1. Checks expected_identity, the identity the files were produced under, before loading anything. It raises AnnotationIdentityError when that identity differs from the source's, or when the source declares none. A scoped source also raises when you leave expected_identity out, because it would stamp its own identity on files an outdated annotation image produced. Pass verify_identity=False only if you checked the files' identity another way.
  2. Loads the files into a scratch table in the annotation table's dataset, or in staging_dataset=. The scratch table has the declared columns' types, so repeated and struct columns load natively, and fields that are not declared columns are ignored. The table expires after six hours (expiration=) and is dropped when the call returns or fails.
  3. Checks every row, and inserts nothing if any row fails:
    • The variant_id must be canonical, with different alleles. Otherwise it raises VariantIdentityError.
    • A row's own chr, pos, ref and alt fields must name the same variant as its variant_id. pos must be equal, the alleles may differ in case, and chr may lack the chr prefix or spell chrM as MT. Otherwise it raises VariantIdentityError. A row without these fields is not checked.
    • On a scoped table, a row's own _altar_annotation_identity field must be empty or the source's identity hash. Otherwise it raises AnnotationIdentityError.
  4. Inserts one row per variant_id the table lacks for this build and, on a scoped table, this identity. It writes genome_default, the chr, pos, ref and alt parsed from the variant_id, and on a scoped table the identity hash. Rows of one variant within the load collapse to one, the first by the row's JSON text, so loading the same files again adds nothing.

It returns InsertResult(added=..., skipped=...). skipped counts the loaded rows not inserted: variants the table already had, and repeated rows within the load. The source needs genome_default, and the table needs ensure_schema() first. The check for variants the table already has reads the variant_id, genome and identity columns of every row of the build, about 8.6 GB per load on an 80-million-row table.

You run the annotation jobs and read the sidecar. Run one load at a time per table: two concurrent loads of the same variants can each insert a row, as with the other write paths.

Write a relation over a scoped table

Altar cannot add a filter to SQL you own. A scoped source's relation= must contain the {identity_filter} token, which expands to _altar_annotation_identity = '<identity_hash>'. A scoped source whose relation lacks the token raises ValueError, and so does an unscoped source whose relation uses it.

reader = BigQueryAnnotationSource.connect(
    "my-project.annotations.variant_regions",
    list(REGION_ANNOTATION_CONTRACT.columns),
    name="regions",
    relation=(
        "(SELECT variant_id, region_type FROM `my-project.annotations.variant_regions` "
        "WHERE {identity_filter} AND {variant_filter})"
    ),
    identity=REGION_ANNOTATION_CONTRACT.identity(release="gencode-v47-screen-v4", genome_build="hg38"),
    identity_scoped=True,
)

A reader needs no genome_default; only writes and deduplication use it.

Altar does not check reads through an unscoped relation=, because only your SQL names the tables it reads. Writes and deduplication always use the source's own table, so Altar checks those. A relation over a scoped table without {identity_filter} reads every release's rows, with no warning. Review each relation that reads a scoped table: scope its source and add the token.

Scope a SQLite table

A scoped SqliteAnnotationSource records its identity, with its hash and readable fields, in the file's _altar_annotation_identity table the first time it opens a file with no rows. A scoped source with a different identity cannot open the file and raises AnnotationIdentityError, so a new release starts a new file. An unscoped source can read any file, but cannot write to a file that records an identity.

A table that already holds rows written without an identity, in SQLite or BigQuery, needs a one-time backfill before a scoped source serves them; see Adopt an annotation table written before source identities.

Roll out a new release

Bump the writer and the readers in two steps:

  1. Change release on the writer. Its next deduplication reports every variant as unannotated. Run jobs until it reports none for the variants you serve. The readers keep serving the old release meanwhile.
  2. Then change release on every reader.

A reader bumped before re-annotation finishes serves nothing for the variants not yet annotated under the new release. Their annotation columns are empty, so a prioritization rule that reads them, such as ChromBPNet's region_type, leaves those variants unprioritized without a warning.

Combine your sources with shipped ones

Pass your own sources and the shipped ones together. Column names must be unique across all sources.

async def assemble(store, variant_ids, models, regulatory, alphamissense):
    async for result in store.materialize(
        variant_ids=variant_ids,
        models=models,
        annotation_sources=[regulatory, alphamissense],
    ):
        print(result.variant_id, result.prioritized, result.annotations)

On BigQueryScoreStore, a BigQueryAnnotationSource on the same project joins into the store's query, and a source on another backend is read through annotate(); see Fused sources.

Close your sources

A source you build in code stays yours. Pass it to altar.run.run as annotation_sources, or to materialize, and the run leaves it open, so you can pass it to the next run and close it when you are done. SqliteAnnotationSource holds a connection; close it, or use it in a with block:

import asyncio

from altar.run import run
from altar.sources import AnnotationColumn, SqliteAnnotationSource

columns = [
    AnnotationColumn("regulatory_class", "str", "Regulatory class"),
    AnnotationColumn("regulatory_score", "float", "Regulatory score"),
]
with SqliteAnnotationSource("regulatory.sqlite", columns, name="regulatory") as regulatory:
    summary = asyncio.run(run("annotate.toml", annotation_sources=[regulatory]))

A source closes only what it opened. A source that takes a client it did not create keeps the default close(), which does nothing, and whoever created the client closes it. BigQueryAnnotationSource.connect creates its own client, so that source's close() closes the client. altar run closes the sources it builds from [sources.<name>] sections once, when the run ends, including when it fails or is cancelled.

Annotation dependencies

A model's prioritization rule can read annotation columns. Its manifest declares them as AnnotationDependency values. Each dependency names a contract by source_id, a minimum schema_version, and the columns the rule reads. ChromBPNet depends on region_type from org.kundajelab.altar.annotation.regions, which altar.variants publishes as REGION_ANNOTATION_CONTRACT.

Before it evaluates any rule, materialize checks each dependency against the configured sources with resolve_annotation_dependencies. The BigQuery exports that compute prioritized run the same check:

  1. If a source's identity names the dependency's contract, that source must declare every dependency column. Its contract version must have the same major version as the dependency and be no older. Only one source may declare that contract. When Altar knows the contract, each dependency column must also have the contract's logical type: the same dtype, repeated flag and nested fields; display labels may differ. Otherwise AnnotationDependencyError is raised, naming the column, the type the source declares, and the type the contract defines.
  2. If no source declares the contract, but other sources declare every dependency column, the columns are used by name. Altar emits an UndeclaredAnnotationContractWarning that names the models and the contract.
  3. If a dependency column is missing from every source, AnnotationDependencyError is raised, naming the contract and the missing columns. The exception is a dependency no source can supply on the variants' assembly, described next.

Sources and variants are matched by assembly. The variants' assembly is the genome build the models were scored on. applicable_annotation_sources keeps the sources that serve it and drops the rest: a source whose identity declares another assembly, and a source with no identity when the variants are not on hg38. Only the kept sources take part in the steps above.

Whether a drop warns or raises depends on whether a model needs the dropped source and whether a source on the variants' assembly could exist. Altar asks the source classes that implement the dependency's contract: the altar.annotation_sources entry points plus the classes of the sources you pass, whose annotation_contract() names it, and their supported_genome_builds (None serves any assembly):

Situation Result
A dropped source no model's dependency needs Dropped, named in an AnnotationUnavailableWarning
A dependency the kept sources do not meet, and classes implement its contract but none serves the variants' assembly Unavailable: null columns, AnnotationUnavailableWarning
A dependency only dropped sources supply, and a class serves the variants' assembly or none implements the contract AnnotationGenomeBuildError: point a source at the variants' assembly
Nothing supplies a dependency otherwise AnnotationDependencyError (step 3)

For example, region data exists only for hg38, and RegionAnnotationSource declares supported_genome_builds = {"hg38"}. A ChromBPNet model that scored mouse variants therefore materializes with a null region_type, whether you pass no sources or your human region table. An hg19 region table as the only region source for hg38 variants raises. A contract no installed class implements, such as your own, could be served on any assembly, so a copy on the wrong assembly that a model needs raises. Declare annotation_contract() and supported_genome_builds on your source class to say its data does not exist elsewhere. Installing a source package that serves another assembly can turn a warning into an error, because the data then exists and should be configured.

Models scored on different assemblies cannot be materialized together when any source is configured. With no models, every source that declares an identity must declare the same assembly. Otherwise AnnotationGenomeBuildError is raised.

Altar knows the contracts it publishes, such as REGION_ANNOTATION_CONTRACT. It does not know a binding's contract or yours, so for those step 1 checks column names only. To check your own contract's column types too, pass it when you call the resolver: resolve_annotation_dependencies(models, sources, contracts=[REGULATORY]). Materialization does not take contracts. To check a source against every column of its contract, use altar.testing.assert_implements_contract(source, contract).

To clear UndeclaredAnnotationContractWarning, declare the contract on the source that supplies the columns: pass identity= to a generic source, or override identity(). For ChromBPNet's dependency, declare the region contract on the table that holds the region columns. Genomic regions explains which release to name:

from altar.sources import BigQueryAnnotationSource
from altar.variants import REGION_ANNOTATION_CONTRACT

regions = BigQueryAnnotationSource.connect(
    "my-project.annotations.variant_regions",
    REGION_ANNOTATION_CONTRACT.columns,
    name="regions",
    genome_default="GRCh38",
    identity=REGION_ANNOTATION_CONTRACT.identity(
        release="ensembl-116.ccre-v4.genes-gencode-47.logic-1", genome_build="hg38"
    ),
)

For your own contracts, name a release that changes whenever the values can change, such as the upstream releases the table was built from. If Altar also writes the table, add identity_scoped=True so a release change re-annotates it, and adopt its existing rows first. Call resolve_annotation_dependencies(models, sources) yourself to check a configuration before you submit work. It returns one ResolvedAnnotationDependency per model dependency, with the sources that satisfied it.

Next