Serve annotations from your own tables¶
Your own application can serve annotations from tables it maintains and combine them with the
annotation sources Altar ships, such as AlphaMissense. A source over your own table is an
ordinary AnnotationSource that you pass to materialize or to altar.run.run. It needs no package, registry entry or
entry point.
Altar checks, joins and prioritizes on what a source declares: its columns, from annotation_columns(), and its
identity, from identity(). It never looks at the source's class or registry name. To write a source class, see
Add an annotation source. To test one with the conformance suite, see
Validate an implementation.
Everything here is in altar.sources and is Stable, except BigQueryAnnotationSource, which is
Preview; see Run at warehouse scale on BigQuery.
Declare a contract and an identity¶
A table you serve implements an AnnotationContract: a published, versioned set of columns named by a source_id.
contract.identity(release=..., genome_build=...) gives the AnnotationSourceIdentity a source declares: the contract,
the data release, and the assembly of the variant IDs it answers for. Contracts and
identity gives the naming and versioning rules. The examples on
this page use this contract:
from altar.sources import AnnotationColumn, AnnotationContract
REGULATORY = AnnotationContract(
source_id="org.example.annotation.regulatory",
schema_version="1.0.0",
columns=(
AnnotationColumn("regulatory_class", "str", "Regulatory class"),
AnnotationColumn("regulatory_score", "float", "Regulatory score"),
),
)
identity = REGULATORY.identity(release="2026-09", genome_build="hg38")
Change release whenever the values the table returns could change, such as after a new upstream download.
identity.identity_hash cannot be turned back into the release; to find the hash of a release, build its identity and
read identity_hash. Declare an identity on every source, because materialization joins a source only to variants on
its assembly (Assemblies). To serve a shipped binding's columns from your
own table, implement its contract, such as ALPHAMISSENSE_ANNOTATIONS from altar_alphamissense or
altar.variants.REGION_ANNOTATION_CONTRACT.
Configure a generic source over your table¶
When a table already has one row per variant_id, configure BigQueryAnnotationSource or SqliteAnnotationSource
instead of writing a class. Pass identity= to declare the contract the table implements:
from altar.sources import BigQueryAnnotationSource
regulatory = BigQueryAnnotationSource.connect(
"my-project.annotations.regulatory_v2026_09",
list(REGULATORY.columns),
name="regulatory",
identity=REGULATORY.identity(release="2026-09", genome_build="hg38"),
)
connect builds a BigQuery client from ambient credentials and needs the altar[bigquery] extra. To pass a client you
built, call the constructor, BigQueryAnnotationSource(client, table_id, columns, ...). A relation= subquery selects,
renames or aggregates the table's columns; see Fused sources.
On its own, identity= is metadata. It does not change how a source writes, deduplicates or reads rows. To make the
identity decide which rows a table serves and counts, also pass identity_scoped=True; see
Keep a writable annotation table current across releases.
identity.genome_build and genome_default are separate settings. identity.genome_build is the build label that
dependency resolution compares with your models' runs, such as hg38. genome_default is how a writable table spells
the build in its stored genome column. Altar's VCF ingest writes GRCh38 there. A table whose rows say GRCh38
keeps genome_default="GRCh38" and can still declare an hg38 identity. Neither setting fills in the other.
A source that reads a dataset in object storage or a warehouse can take a named connection; see Pass credentials through connections. The connection is not part of the identity, so one release read two ways keeps one identity.
Keep a writable annotation table current across releases¶
A writable annotation table is a table Altar writes annotations into and then reads back as a cache, such as the
region labels your annotation jobs compute. Pass identity_scoped=True with identity= to keep each release's rows
apart. Both generic sources take the flag:
- A scoped
BigQueryAnnotationSourcestores the identity's hash on every row and uses only rows with that hash. - A scoped
SqliteAnnotationSourcerecords one identity per file and refuses to open the file under another.
In both, changing release makes every variant a cache miss, and your pipeline annotates them again. Without the flag,
identity= is metadata on both.
A scoped BigQueryAnnotationSource works like this:
ensure_schema()adds a nullable_altar_annotation_identity STRINGcolumn. The name isaltar.sources.ANNOTATION_IDENTITY_COLUMN.add_annotations,add_annotations_frameandadd_annotations_from_uriwrite the identity'sidentity_hashinto that column on every row. A row that already carries a different hash raisesAnnotationIdentityError.get_unannotated_variantsandexport_unannotatedcount a variant as annotated only when it has a row with that hash.annotateand materialization read only rows with that hash, so a table that holds several releases still joins one row per variant.
from altar.sources import BigQueryAnnotationSource
from altar.variants import REGION_ANNOTATION_CONTRACT
regions = BigQueryAnnotationSource.connect(
"my-project.annotations.variant_regions",
list(REGION_ANNOTATION_CONTRACT.columns),
name="regions",
genome_default="GRCh38",
identity=REGION_ANNOTATION_CONTRACT.identity(release="gencode-v47-screen-v4", genome_build="hg38"),
identity_scoped=True,
)
async def unannotated_variants():
await regions.ensure_schema()
return await regions.get_unannotated_variants([], variants_relation="`my-project.jobs.variants_123`")
The examples below use this regions source.
Every write stores the row's genome and its chr, pos, ref and alt beside the canonical variant_id, and
get_unannotated_variants counts only rows of the source's genome_default build.
Scope every source object that reads or writes the table, with the same identity. A reader you configure separately
from the writer needs identity_scoped=True too. Otherwise it reads every release's rows, and a variant can be served
any release's values.
An unscoped BigQueryAnnotationSource checks for this. It reads its table's schema, a metadata request that runs no
query and bills nothing. When the table has the _altar_annotation_identity column, the unscoped source:
- refuses to write to it.
add_annotations,add_annotations_frameandadd_annotations_from_uriraiseAnnotationIdentityErrorbefore writing or loading anything, because the table's scoped sources would neither serve nor count rows without the hash. A SQLite table refuses an unscoped write the same way. - warns before its first read or deduplication. Materialization, the BigQuery exports,
annotate(),get_unannotated_variantsandexport_unannotatedemitUnscopedAnnotationReadWarningand then run. An unscoped deduplication counts a variant annotated under any release as annotated. Each source object warns at most once.
If the schema lookup fails, for example for lack of permission, Altar logs it at debug level and runs unchecked. To make the warning fail a job, turn it into an error:
import warnings
from altar.sources import UnscopedAnnotationReadWarning
warnings.simplefilter("error", UnscopedAnnotationReadWarning)
Load annotation files from object storage¶
export_unannotated writes an annotation job's input. For each variant in your job relation that the table has no row
for, it writes a headerless, tab-separated line of chr, pos, ref, alt, variant_id to object storage, and it returns
how many lines it wrote. The first four columns are split from the variant_id, so the relation needs only a
variant_id column. The region annotate command copies the fifth column into each JSONL row's variant_id, which the
load below requires.
async def export_job_input():
return await regions.export_unannotated(
destination_uri="gs://my-bucket/run-7/unannotated-*.tsv",
variants_relation="`my-project.jobs.variants_123`",
)
export_unannotated and get_unannotated_variants never report a variant whose variant_id is not canonical, such as
chr9:21980945:C:C (REF equal to ALT) or 9:100:C:T, because no write path can store one. export_unannotated logs a
warning with how many it left out and an example. Delete them from the tables your job relation reads.
BigQuery's EXPORT DATA may write several files under the wildcard, and some may be empty. The annotate command reads
one file, so each job runs these steps:
- Export to a fresh prefix for this job, such as
gs://my-bucket/run-7/. The export replaces only the files it writes. It never deletes stale files an earlier export left under the same wildcard, and an export that finds nothing to annotate writes no file. Reusing a prefix can therefore annotate an earlier job's variants again. - Annotate each
unannotated-*.tsvfile into its own JSONL, such asannotated-000000000000.jsonl. An empty file gives an empty JSONL. You can instead concatenate the files and annotate one. - Load every JSONL of the job with one call to
add_annotations_from_uri, passing oneexpected_identity. Every file's identity sidecar names the same identity when one annotate image produced them.
from altar.sources import load_annotation_identity
async def load_job_output(sidecar_bytes: bytes):
# sidecar_bytes: gs://my-bucket/run-7/annotated.jsonl.identity.json, read with your own storage client.
produced_under = load_annotation_identity(sidecar_bytes)
result = await regions.add_annotations_from_uri("gs://my-bucket/run-7/*.jsonl", expected_identity=produced_under)
print(result.added, result.skipped)
load_annotation_identity reads the identity sidecar the annotate command writes next to its JSONL. It takes the
sidecar's text or bytes, its parsed object, or a local pathlib.Path, and checks the sidecar's format and that its
identity_hash matches its identity. Altar does not read object storage here, so read the sidecar with your own client.
add_annotations_from_uri takes any URI, or list of URIs, a BigQuery load job accepts. It:
- Checks
expected_identity, the identity the files were produced under, before loading anything. It raisesAnnotationIdentityErrorwhen that identity differs from the source's, or when the source declares none. A scoped source also raises when you leaveexpected_identityout, because it would stamp its own identity on files an outdated annotation image produced. Passverify_identity=Falseonly if you checked the files' identity another way. - Loads the files into a scratch table in the annotation table's dataset, or in
staging_dataset=. The scratch table has the declared columns' types, so repeated and struct columns load natively, and fields that are not declared columns are ignored. The table expires after six hours (expiration=) and is dropped when the call returns or fails. - Checks every row, and inserts nothing if any row fails:
- The
variant_idmust be canonical, with different alleles. Otherwise it raisesVariantIdentityError. - A row's own
chr,pos,refandaltfields must name the same variant as itsvariant_id.posmust be equal, the alleles may differ in case, andchrmay lack thechrprefix or spellchrMasMT. Otherwise it raisesVariantIdentityError. A row without these fields is not checked. - On a scoped table, a row's own
_altar_annotation_identityfield must be empty or the source's identity hash. Otherwise it raisesAnnotationIdentityError.
- The
- Inserts one row per
variant_idthe table lacks for this build and, on a scoped table, this identity. It writesgenome_default, thechr,pos,refandaltparsed from thevariant_id, and on a scoped table the identity hash. Rows of one variant within the load collapse to one, the first by the row's JSON text, so loading the same files again adds nothing.
It returns InsertResult(added=..., skipped=...). skipped counts the loaded rows not inserted: variants the table
already had, and repeated rows within the load. The source needs genome_default, and the table needs ensure_schema()
first. The check for variants the table already has reads the variant_id, genome and identity columns of every
row of the build, about 8.6 GB per load on an 80-million-row table.
You run the annotation jobs and read the sidecar. Run one load at a time per table: two concurrent loads of the same variants can each insert a row, as with the other write paths.
Write a relation over a scoped table¶
Altar cannot add a filter to SQL you own. A scoped source's relation= must contain the {identity_filter} token,
which expands to _altar_annotation_identity = '<identity_hash>'. A scoped source whose relation lacks the token raises
ValueError, and so does an unscoped source whose relation uses it.
reader = BigQueryAnnotationSource.connect(
"my-project.annotations.variant_regions",
list(REGION_ANNOTATION_CONTRACT.columns),
name="regions",
relation=(
"(SELECT variant_id, region_type FROM `my-project.annotations.variant_regions` "
"WHERE {identity_filter} AND {variant_filter})"
),
identity=REGION_ANNOTATION_CONTRACT.identity(release="gencode-v47-screen-v4", genome_build="hg38"),
identity_scoped=True,
)
A reader needs no genome_default; only writes and deduplication use it.
Altar does not check reads through an unscoped relation=, because only your SQL names the tables it reads. Writes and
deduplication always use the source's own table, so Altar checks those. A relation over a scoped table without
{identity_filter} reads every release's rows, with no warning. Review each relation that reads a scoped table: scope
its source and add the token.
Scope a SQLite table¶
A scoped SqliteAnnotationSource records its identity, with its hash and readable fields, in the file's
_altar_annotation_identity table the first time it opens a file with no rows. A scoped source with a different
identity cannot open the file and raises AnnotationIdentityError, so a new release starts a new file. An unscoped
source can read any file, but cannot write to a file that records an identity.
A table that already holds rows written without an identity, in SQLite or BigQuery, needs a one-time backfill before a scoped source serves them; see Adopt an annotation table written before source identities.
Roll out a new release¶
Bump the writer and the readers in two steps:
- Change
releaseon the writer. Its next deduplication reports every variant as unannotated. Run jobs until it reports none for the variants you serve. The readers keep serving the old release meanwhile. - Then change
releaseon every reader.
A reader bumped before re-annotation finishes serves nothing for the variants not yet annotated under the new release.
Their annotation columns are empty, so a prioritization rule that reads them, such as ChromBPNet's region_type, leaves
those variants unprioritized without a warning.
Combine your sources with shipped ones¶
Pass your own sources and the shipped ones together. Column names must be unique across all sources.
async def assemble(store, variant_ids, models, regulatory, alphamissense):
async for result in store.materialize(
variant_ids=variant_ids,
models=models,
annotation_sources=[regulatory, alphamissense],
):
print(result.variant_id, result.prioritized, result.annotations)
On BigQueryScoreStore, a BigQueryAnnotationSource on the same project joins into the store's query, and a source on
another backend is read through annotate(); see Fused sources.
Close your sources¶
A source you build in code stays yours. Pass it to altar.run.run as annotation_sources, or to materialize, and the
run leaves it open, so you can pass it to the next run and close it when you are done. SqliteAnnotationSource holds a
connection; close it, or use it in a with block:
import asyncio
from altar.run import run
from altar.sources import AnnotationColumn, SqliteAnnotationSource
columns = [
AnnotationColumn("regulatory_class", "str", "Regulatory class"),
AnnotationColumn("regulatory_score", "float", "Regulatory score"),
]
with SqliteAnnotationSource("regulatory.sqlite", columns, name="regulatory") as regulatory:
summary = asyncio.run(run("annotate.toml", annotation_sources=[regulatory]))
A source closes only what it opened. A source that takes a client it did not create keeps the default close(), which
does nothing, and whoever created the client closes it. BigQueryAnnotationSource.connect creates its own client, so
that source's close() closes the client. altar run closes the sources it builds from [sources.<name>] sections
once, when the run ends, including when it fails or is cancelled.
Annotation dependencies¶
A model's prioritization rule can read annotation columns. Its manifest declares them as AnnotationDependency values.
Each dependency names a contract by source_id, a minimum schema_version, and the columns the rule reads. ChromBPNet
depends on region_type from org.kundajelab.altar.annotation.regions, which altar.variants publishes as
REGION_ANNOTATION_CONTRACT.
Before it evaluates any rule, materialize checks each dependency against the configured sources with
resolve_annotation_dependencies. The BigQuery exports that compute prioritized run the same check:
- If a source's identity names the dependency's contract, that source must declare every dependency column. Its
contract version must have the same major version as the dependency and be no older. Only one source may declare that
contract. When Altar knows the contract, each dependency column must also have the contract's logical type: the same
dtype,repeatedflag and nestedfields; display labels may differ. OtherwiseAnnotationDependencyErroris raised, naming the column, the type the source declares, and the type the contract defines. - If no source declares the contract, but other sources declare every dependency column, the columns are used by name.
Altar emits an
UndeclaredAnnotationContractWarningthat names the models and the contract. - If a dependency column is missing from every source,
AnnotationDependencyErroris raised, naming the contract and the missing columns. The exception is a dependency no source can supply on the variants' assembly, described next.
Sources and variants are matched by assembly. The variants' assembly is the genome build the models were scored on.
applicable_annotation_sources keeps the sources that serve it and drops the rest: a source whose identity declares
another assembly, and a source with no identity when the variants are not on hg38. Only the kept sources take part in
the steps above.
Whether a drop warns or raises depends on whether a model needs the dropped source and whether a source on the variants'
assembly could exist. Altar asks the source classes that implement the dependency's contract: the
altar.annotation_sources entry points plus the classes of the sources you pass, whose annotation_contract() names
it, and their supported_genome_builds (None serves any assembly):
| Situation | Result |
|---|---|
| A dropped source no model's dependency needs | Dropped, named in an AnnotationUnavailableWarning |
| A dependency the kept sources do not meet, and classes implement its contract but none serves the variants' assembly | Unavailable: null columns, AnnotationUnavailableWarning |
| A dependency only dropped sources supply, and a class serves the variants' assembly or none implements the contract | AnnotationGenomeBuildError: point a source at the variants' assembly |
| Nothing supplies a dependency otherwise | AnnotationDependencyError (step 3) |
For example, region data exists only for hg38, and RegionAnnotationSource declares
supported_genome_builds = {"hg38"}. A ChromBPNet model that scored mouse variants therefore materializes with a null
region_type, whether you pass no sources or your human region table. An hg19 region table as the only region source
for hg38 variants raises. A contract no installed class implements, such as your own, could be served on any assembly,
so a copy on the wrong assembly that a model needs raises. Declare annotation_contract() and
supported_genome_builds on your source class to say its data does not exist elsewhere. Installing a source package
that serves another assembly can turn a warning into an error, because the data then exists and should be configured.
Models scored on different assemblies cannot be materialized together when any source is configured. With no models,
every source that declares an identity must declare the same assembly. Otherwise AnnotationGenomeBuildError is
raised.
Altar knows the contracts it publishes, such as REGION_ANNOTATION_CONTRACT. It does not know a binding's contract or
yours, so for those step 1 checks column names only. To check your own contract's column types too, pass it when you
call the resolver: resolve_annotation_dependencies(models, sources, contracts=[REGULATORY]). Materialization does not
take contracts. To check a source against every column of its contract, use
altar.testing.assert_implements_contract(source, contract).
To clear UndeclaredAnnotationContractWarning, declare the contract on the source that supplies the columns: pass
identity= to a generic source, or override identity(). For ChromBPNet's dependency, declare the region contract on
the table that holds the region columns. Genomic regions explains
which release to name:
from altar.sources import BigQueryAnnotationSource
from altar.variants import REGION_ANNOTATION_CONTRACT
regions = BigQueryAnnotationSource.connect(
"my-project.annotations.variant_regions",
REGION_ANNOTATION_CONTRACT.columns,
name="regions",
genome_default="GRCh38",
identity=REGION_ANNOTATION_CONTRACT.identity(
release="ensembl-116.ccre-v4.genes-gencode-47.logic-1", genome_build="hg38"
),
)
For your own contracts, name a release that changes whenever the values can change, such as the upstream releases the
table was built from. If Altar also writes the table, add identity_scoped=True so a release change re-annotates it,
and adopt its existing rows
first. Call resolve_annotation_dependencies(models, sources) yourself to check a configuration before you submit work.
It returns one ResolvedAnnotationDependency per model dependency, with the sources that satisfied it.
Next¶
- Run at warehouse scale on BigQuery joins BigQuery tables into the score store's query.
- Add an annotation source writes a source class when a generic source is not enough.