Genomic regions¶
The regions source adds where each hg38 variant falls: its gene region (coding, UTR, splice, intron, promoter or
intergenic), the genes it overlaps, the nearest protein-coding genes and an overlapping ENCODE cCRE. It computes
these from region files you build once from Ensembl, SCREEN and GENCODE. It runs no model.
| Kind | Annotation source; type = "regions" |
| Install | pip install altar (core) (PyPI) |
| Genome builds | hg38 |
| Variant classes | Every class |
| Data | Bring your own |
| Validation | Not spot-checked (Validation) |
| Licence | MIT code; open Ensembl, SCREEN and GENCODE data |
| Upstream | Data, Documentation |
When to use it¶
- Use the regions source to ask where a variant sits relative to genes and candidate regulatory elements: coding, splice, UTR, intron, promoter or intergenic, which genes are nearest, and whether it falls in a SCREEN cCRE.
- Use it to send each variant to the scorer that fits its region (Route variants by region).
- A run that scores hg38 variants with ChromBPNet needs it: ChromBPNet's prioritization rule
reads
region_type(Prioritization). - Choose AlphaMissense or SpliceAI for coding and splice effects. The regions source labels a coding or splice-site variant but computes no codon-level consequence, such as missense.
- Choose Open Targets E2G or the ENCODE-rE2G and scE2G atlases for the genes a regulatory element is predicted to regulate. The nearest gene is a distance, not a predicted target.
Add it to a run¶
- The source is part of core Altar:
pip install altar. - Build the region files once (Build another release), or run in the variants runtime image, which contains them.
-
Name the directory that holds them as
data_dir, or setALTAR_VARIANTS_DATA_DIR:
altar sources regions lists the columns and options. The run spec reference
describes the [sources.<name>] section, and Annotate variants with evidence walks
through a run.
Outputs¶
| Column | Type | Meaning |
|---|---|---|
region_type |
str | The most severe region label, such as cds, splice_site, promoter or intergenic. |
region_labels |
list of str | Every overlapping region label, most severe first. |
region_gene_ids |
list of str | Ensembl IDs of the genes whose features the variant overlaps. |
is_coding |
bool | The variant overlaps a CDS. |
in_promoter |
bool | The variant overlaps the promoter window of any gene. |
nearest_genes |
list of struct (gene_name, gene_id, distance, type) |
The five nearest protein-coding genes, each with its name, ID, distance and gene type. |
gene_within_100kb |
bool | The nearest protein-coding gene lies within 100 kb. |
ccre_id |
str | Accession of an overlapping SCREEN cCRE; null when none. |
ccre_group |
str | That cCRE's group, such as PLS or dELS; null when none. |
The labels and windows are defined under Region classification. region_type is
promoter only when no more severe label applies; in_promoter is true for any promoter overlap.
Each nearest_genes record has gene_name, gene_id, distance and type. distance is 0 inside a gene.
Outside one, it is the distance in bases to the gene's nearer end, negative when the variant lies at a lower
coordinate than the gene. When several cCREs overlap the variant, ccre_id names one of them.
The columns come from two gene annotations. region_type, region_labels, region_gene_ids, is_coding and
in_promoter come from the Ensembl GFF3 the region files were built from (Ensembl 116 by default).
nearest_genes and gene_within_100kb come from the gene table that ships with Altar, GENCODE 47 (Ensembl 113).
A gene can therefore appear in one and not the other, and the two can name a gene differently.
A variant on a chromosome with no protein-coding gene in the gene table, such as an unplaced contig, has no row.
The annotate command writes these columns and some fields outside the source; see The annotate command.
Configuration¶
| Option | Set in | Type | Default | Meaning |
|---|---|---|---|---|
data_dir |
[sources.regions.options] |
path | not set | the directory that holds the region files (default: $ALTAR_VARIANTS_DATA_DIR, else the installed package's data directory) |
genome_build |
[sources.regions.options] |
text | "hg38" |
the genome build of the region files, which must be hg38 |
Both options are optional. Altar looks for the region files in this order:
- the
data_diroption or argument, or--data_dirof the annotate andregion_filtercommands (--data-dirofaltar.variants.preprocessing.pipeline); - the
ALTAR_VARIANTS_DATA_DIRenvironment variable; - the
datadirectory of the installedaltar.variantspackage.
genome_build must be hg38, the default; GRCh38 names the same build. Another option stops the run with an
error that names it.
Inputs¶
- Genome builds: hg38 only. A run on another build stops before it reads the cohort, and the error names the source.
- Variant classes: every class. The source looks each variant up at the reference span of its minimal representation. Its region labels and cCRE come from that span, and its nearest genes from the span's first base.
- Chromosomes: a variant on a chromosome with no protein-coding gene in the gene table, such as an unplaced contig, has no row.
- Region files:
region_annotations.parquet,ccres.dnatreeandregion_provenance.jsonin the data directory, and the shippedgene_df.tsvunless the directory holds its own.
Where it comes from¶
| File | Built from | Size |
|---|---|---|
region_annotations.parquet |
the Ensembl 116 GRCh38 GFF3 | about 32 million intervals |
ccres.dnatree |
the SCREEN Registry v4 GRCh38 cCRE BED | about 2.3 million elements |
gene_df.tsv |
GENCODE 47 (Ensembl 113) gene records; ships with Altar | 78,724 genes |
variants.pkl.gz |
the Open Targets Platform 25.03 variant shards (about 3.1 GB); read only by the annotate command | about 320 MB |
The altar wheel does not include the first two files or variants.pkl.gz. A data directory may hold its own
gene_df.tsv, which then replaces the shipped one for that directory.
gene_df.tsv holds the gene records of GENCODE 47's gencode.v47.annotation.gtf, in file order: the gene name,
chromosome, start and end ordered 5′ to 3′, strand, gene type and the unversioned gene ID. Its SHA-256 is
52807cc8f3e74645d1a2ebc27b95081fc031a56d766aa3f31e3114c5315cc370.
A missing file raises AnnotationDataNotFoundError, a FileNotFoundError. The message names the file, the
directory searched and the command that builds the file.
The variants runtime image, built from runtimes/variants in the repository, contains all the files; see its
README.
Release and identity¶
The source's identity is the org.kundajelab.altar.annotation.regions contract, version 1.0.0, on hg38. Its
release names the three inputs the values come from and the version of the logic that turns them into columns:
ensembl-116: the Ensembl GFF3region_annotations.parquetwas built from.ccre-v4: the SCREEN cCRE Registry BEDccres.dnatreewas built from.genes-gencode-47: the nearest-gene table.logic-1: the region logic version, which covers the build windows and the lookups. A change to either that changes any value bumps it.
Altar pins the SHA-256 of the Ensembl 116 GFF3, the SCREEN v4 BED and the GENCODE 47 gene table. When every input
has its pinned digest, the string above is the whole release. Otherwise, for example for another Ensembl release
or your own gene table, . and the first 12 hex characters of a digest over the three input digests follow. A
gene table other than the shipped one is recorded with release unrecorded.
The release does not depend on the derived files' bytes: files written by another pandas or pyarrow version name the same release. A run records the identity in its provenance.
The region files' directory holds region_provenance.json, which records each input's release and SHA-256, the
logic version and each derived file's SHA-256:
{
"format": 2,
"genome_build": "hg38",
"logic_version": 1,
"inputs": {
"ensembl_gff3": {"source": "Ensembl GRCh38 GFF3", "release": "116", "sha256": "…"},
"ccre_bed": {"source": "SCREEN cCRE Registry GRCh38", "release": "v4", "sha256": "…"},
"gene_table": {"source": "GENCODE GRCh38 gene annotation", "release": "gencode-47", "sha256": "…"}
},
"derived": {
"region_annotations.parquet": {"sha256": "…", "logic_version": 1},
"ccres.dnatree": {"sha256": "…", "logic_version": 1}
}
}
Before the source loads any file, it checks each file's SHA-256 against the manifest. A manifest that no longer describes its directory is an error. A manifest from an earlier Altar, or from another logic version, is refused; rebuild the files.
Build another release¶
Set the data directory first. The download scripts and build commands write under it.
export ALTAR_VARIANTS_DATA_DIR="$HOME/altar-variants-data"
# Region annotations (Ensembl 116 GFF3)
bash altar/altar/variants/scripts/download_ensembl_gff.sh
python -m altar.variants.scripts.construct_region_annotations \
-i "$ALTAR_VARIANTS_DATA_DIR/raw/Homo_sapiens.GRCh38.116.gff3.gz"
# cCRE index (SCREEN Registry v4)
bash altar/altar/variants/scripts/download_ccres.sh
python -m altar.variants.scripts.construct_ccre_dnatree
The download scripts are in altar/altar/variants/scripts/ of the repository, and in altar/variants/scripts/
of the installed package. A missing-file error prints the installed scripts' paths. They need wget. After the
build, $ALTAR_VARIANTS_DATA_DIR/raw is no longer needed.
construct_region_annotations reads the Ensembl release from the GFF3 file name, or from --ensembl-release.
construct_ccre_dnatree records SCREEN release v4, or the one --release names. Each records its input and the
input's SHA-256 in region_provenance.json. Ensembl 116 and SCREEN v4 are pinned: a file labelled with one of
these releases must have the pinned SHA-256, and the command refuses before it builds anything otherwise.
For another Ensembl release, download it with ENSEMBL_RELEASE=<release> set, and build from that GFF3. The
release string then names it and carries the digest suffix.
The annotate command also reads variants.pkl.gz. Building it needs rsync and about 7 GB of memory:
bash altar/altar/variants/scripts/download_variants.sh
python -m altar.variants.scripts.construct_variants_df
To produce the same table as the runtime image, follow runtimes/variants/README.md in the repository.
To record files built by this Altar version from inputs you still have:
python -m altar.variants.annotation.provenance \
--ensembl-release 116 --ensembl-gff "$ALTAR_VARIANTS_DATA_DIR/raw/Homo_sapiens.GRCh38.116.gff3.gz" \
--ccre-release v4 --ccre-bed "$ALTAR_VARIANTS_DATA_DIR/raw/GRCh38-cCREs.bed"
The command takes --data-dir, hashes the inputs and files, writes the manifest and prints the release. A tree
with the runtime image's pinned v4 digest is itself evidence of the v4 BED, so --ccre-bed may be left out for
it. The command never records a release from a label alone. It refuses a pinned release whose input has another
digest, and an input whose bytes are already recorded under a different release.
Compute and scale¶
- A lookup runs no model and needs no GPU. It computes the columns in Python, in the Altar process.
- The first lookup loads the full region index, about 32 million intervals, and the cCRE tree, about 2.3 million elements, into memory. Each variant then costs a scan of its chromosome's genes.
- The source suits tests, notebooks and cohorts that fit in memory. At warehouse scale, run the annotate command and load its output into a table (The annotate command).
- Building
variants.pkl.gzfor the annotate command needs about 7 GB of memory.
Prioritization¶
None. The source never sets prioritized on its own.
ChromBPNet's rule reads region_type: its manifest declares a dependency on the
org.kundajelab.altar.annotation.regions contract, version 1.0.0. A run with ChromBPNet needs a source that
serves this contract, such as regions. See Required annotations,
and What prioritized means for how rules combine.
Region classification¶
The region labels come from interval overlap against the Ensembl gene model. Codon-level consequences, such as missense or stop-gain, are not computed; AlphaMissense and SpliceAI score coding and splice effects.
A variant can carry several labels: a coding base can also be a splice site, and a variant can be exonic in one transcript and intronic in another. The labels, from most to least severe:
| Label | The variant overlaps |
|---|---|
splice_site |
a canonical splice donor or acceptor: the 2 intronic bases next to an exon |
cds |
a coding sequence |
splice_region |
8 intronic or 3 exonic bases around an exon boundary; every splice site is also a splice region |
five_prime_utr, three_prime_utr |
a UTR |
exonic |
an exon of a protein-coding gene |
noncoding_gene |
an exon of a non-coding gene, such as a lncRNA or miRNA |
intronic |
an intron |
promoter |
500 bases upstream to 100 bases downstream of a transcription start site |
intergenic |
none of the above |
Donor and acceptor sites, and 5′ and 3′ UTRs, follow the transcript's strand. Introns, splice windows and
promoters are derived from each transcript's exons when the region files are built. The windows are constants in
altar.variants.scripts.construct_region_annotations.
Route variants by region¶
region_filter splits a variant file into one headerless TSV per region category, so each can go to the scorer
that needs it. A variant is written to every category its labels match, so it can appear in several files.
| File | Membership | Typical scorer |
|---|---|---|
coding.tsv |
overlaps a CDS | AlphaMissense |
exonic.tsv |
protein-coding exon | — |
five_prime_utr.tsv, three_prime_utr.tsv |
UTR | — |
splice.tsv |
splice site or splice region | SpliceAI |
splice_site.tsv |
canonical donor or acceptor | SpliceAI |
splice_region.tsv |
splice region | SpliceAI |
intronic.tsv |
intron | chromatin models |
promoter.tsv |
promoter window | chromatin models |
noncoding_gene.tsv |
non-coding-gene exon | — |
genic.tsv |
any gene-body label (not promoter or intergenic) | SpliceAI |
intergenic.tsv |
no label; written only when asked for | — |
--categories takes a comma-separated subset. --data_dir names the data directory, and -b sets the batch
size (default 250,000). There is no noncoding category: a variant can be coding in one transcript and intronic in
another. Chromatin models such as ChromBPNet score any locus, coding included, so give them the whole variant set.
Classify intervals in Python¶
Provisional
These functions are not part of the public API, and the import below is expected to change.
from altar.variants.annotation.regions import region_annotations, region_annotations_batch
ann = region_annotations("chr1", 69094, 69094)
ann.labels # every label, most severe first
ann.gene_ids # overlapping gene IDs
ann.primary # the most severe label
ann.is_coding # overlaps a CDS
ann.in_promoter # overlaps a promoter window
anns = region_annotations_batch(df["chr"], df["pos"], df["pos"])
Intervals are 1-based and inclusive. Both functions take data_dir=. region_annotations_batch classifies many
intervals in one pass, in input order.
Validation¶
Validation label: Not spot-checked. Altar pins the SHA-256 of each input, refuses a pinned release whose input has
another digest, and checks the derived files against region_provenance.json before it loads them. The labels and
windows are covered by the repository's tests on small synthetic gene models.
Not validated:
- region labels compared with another annotator, such as Ensembl VEP, for named variants;
- nearest genes and cCRE overlaps compared with Ensembl or SCREEN for named variants.
Limits¶
- hg38 only. A run on another build stops before it reads the cohort, and the error names the source.
- The first lookup loads the full region index and cCRE tree into memory (Compute and scale).
- Region labels come from interval overlap. Codon-level consequences, such as missense or stop-gain, are not computed.
- The region labels and the nearest genes come from different gene annotations, which can disagree (Outputs).
- The provenance checks catch drift and accidents, such as a rebuilt file without a rebuilt manifest. They do not protect against someone who can write the data directory, who can rewrite the manifest too.
ccres.dnatreeandvariants.pkl.gzare Python pickles, and loading a pickle can run arbitrary code. The source unpicklesccres.dnatreeonly from bytes whose SHA-256 matches the manifest. Pointdata_dirandALTAR_VARIANTS_DATA_DIRonly at files you built or that come from a source you trust.
Cite and licence¶
Cite the sources of the files you built, and name their releases, such as "Ensembl 116, SCREEN Registry v4, GENCODE 47".
The Ensembl gene model:
@article{dyer2025ensembl,
author = {Dyer, Sarah C and Austine-Orimoloye, Olanrewaju and Azov, Andrey G and Barba, Matthieu and Barnes, If
and Barrera-Enriquez, Vianey Paola and Becker, Arne and Bennett, Ruth and Beracochea, Martin and Berry, Andrew
and Bhai, Jyothish and Bhurji, Simarpreet Kaur and Boddu, Sanjay and Branco Lins, Paulo R and Brooks, Lucy and
Ramaraju, Shashank Budhanuru and Campbell, Lahcen I and Martinez, Manuel Carbajo and Charkhchi, Mehrnaz and
Cortes, Lucas A and Davidson, Claire and Denni, Sukanya and Dodiya, Kamalkumar and Donaldson, Sarah and
El Houdaigui, Bilal and El Naboulsi, Tamara and Falola, Oluwadamilare and Fatima, Reham and Genez, Thiago and
Martinez, Jose Gonzalez and Gurbich, Tatiana and Hardy, Matthew and Hollis, Zoe and Hunt, Toby and Kay, Mike and
Kaykala, Vinay and Lemos, Diana and Lodha, Disha and Mathlouthi, Nourhen and Merino, Gabriela Alejandra and
Merritt, Ryan and Mirabueno, Louisse Paola and Mushtaq, Aleena and Hossain, Syed Nakib and
P{\'e}rez-Silva, Jos{\'e} G and Perry, Malcolm and Pili{\v{z}}ota, Ivana and Poppleton, Daniel and
Prosovetskaia, Irina and Raj, Shriya and Salam, Ahamed Imran Abdul and Saraf, Shradha and
Saraiva-Agostinho, Nuno and Sinha, Swati and Sipos, Botond and Sitnik, Vasily and Steed, Emily and
Suner, Marie-Marthe and Surapaneni, Likhitha and Sutinen, Ky{\"o}sti and Tricomi, Francesca Floriana and
Tsang, Ian and Urbina-G{\'o}mez, David and Veidenberg, Andres and Walsh, Thomas A and Willhoft, Natalie L and
Allen, Jamie and Alvarez-Jarreta, Jorge and Chakiachvili, Marc and Cheema, Jitender and da Rocha, Jorge Batista
and De Silva, Nishadi H and Giorgetti, Stefano and Haggerty, Leanne and Ilsley, Garth R and Keatley, Jon and
Loveland, Jane E and Moore, Benjamin and Mudge, Jonathan M and Naamati, Guy and Tate, John and
Trevanion, Stephen J and Winterbottom, Andrea and Flint, Bethany and Frankish, Adam and Hunt, Sarah E and
Finn, Robert D and Freeberg, Mallory A and Harrison, Peter W and Martin, Fergal J and Yates, Andrew D},
title = {Ensembl 2025},
journal = {Nucleic Acids Research},
volume = {53},
number = {D1},
pages = {D948--D957},
year = {2025},
doi = {10.1093/nar/gkae1071}
}
The SCREEN cCREs:
@article{moore2026screen,
author = {Moore, Jill E. and Pratt, Henry E. and Fan, Kaili and Phalke, Nishigandha and Fisher, Jonathan and
Elhajjajy, Shaimae I. and Andrews, Gregory and Gao, Mingshi and Shedd, Nicole and Fu, Yu and Lacadie, Matthew C.
and Meza, Jair and Khandpekar, Mansi and Ganna, Mohit and Choudhury, Eva and Swofford, Ross and Phan, Huong and
Ramirez, Christian C. and Campbell, Maxwell and Likhite, Mary and Farrell, Nina P. and Weimer, Annika K. and
Pampari, Anusri and Ramalingam, Vivekanandan and Reese, Fairlie and Borsari, Beatrice and Yu, Xuezhu and
Wattenberg, Eve and Ruiz-Romero, Marina and Razavi-Mohseni, Milad and Xu, Jinrui and Galeev, Timur and
Colubri, Andres and Beer, Michael A. and Guig{\'o}, Roderic and Gerstein, Mark B. and Engreitz, Jesse M. and
Ljungman, Mats and Reddy, Timothy E. and Snyder, Michael P. and Epstein, Charles B. and Gaskell, Elizabeth and
Bernstein, Bradley E. and Dickel, Diane E. and Visel, Axel and Pennacchio, Len A. and Mortazavi, Ali and
Kundaje, Anshul and Weng, Zhiping},
title = {An expanded registry of candidate cis-regulatory elements},
journal = {Nature},
year = {2026},
doi = {10.1038/s41586-025-09909-9}
}
The GENCODE gene table:
@article{mudge2025gencode,
author = {Mudge, Jonathan M and Carbonell-Sala, S{\'i}lvia and Diekhans, Mark and Martinez, Jose Gonzalez and
Hunt, Toby and Jungreis, Irwin and Loveland, Jane E and Arnan, Carme and Barnes, If and Bennett, Ruth and
Berry, Andrew and Bignell, Alexandra and Cerd{\'a}n-V{\'e}lez, Daniel and Cochran, Kelly and Cort{\'e}s, Lucas T
and Davidson, Claire and Donaldson, Sarah and Dursun, Cagatay and Fatima, Reham and Hardy, Matthew and
Hebbar, Prajna and Hollis, Zoe and James, Benjamin T and Jiang, Yunzhe and Johnson, Rory and Kaur, Gazaldeep and
Kay, Mike and Mangan, Riley J and Maquedano, Miguel and G{\'o}mez, Laura Mart{\'i}nez and Mathlouthi, Nourhen
and Merritt, Ryan and Ni, Pengyu and Palumbo, Emilio and Perteghella, Tamara and Pozo, Fernando and Raj, Shriya
and Sisu, Cristina and Steed, Emily and Sumathipala, Dulika and Suner, Marie-Marthe and
Uszczynska-Ratajczak, Barbara and Wass, Elizabeth and Yang, Yucheng T and Zhang, Dingyao and Finn, Robert D and
Gerstein, Mark and Guig{\'o}, Roderic and Hubbard, Tim J P and Kellis, Manolis and Kundaje, Anshul and
Paten, Benedict and Tress, Michael L and Birney, Ewan and Martin, Fergal J and Frankish, Adam},
title = {{GENCODE} 2025: reference gene annotation for human and mouse},
journal = {Nucleic Acids Research},
volume = {53},
number = {D1},
pages = {D966--D975},
year = {2025},
doi = {10.1093/nar/gkae1078}
}
The annotate command's af_* frequencies come from the Open Targets Platform. The frequencies are gnomAD's; cite
gnomAD as its page describes, and Open Targets:
@article{buniello2025opentargets,
author = {Buniello, Annalisa and Suveges, Daniel and Cruz-Castillo, Carlos and Llinares, Manuel Bernal and
Cornu, Helena and Lopez, Irene and Tsukanov, Kirill and Rold{\'a}n-Romero, Juan Mar{\'i}a and Mehta, Chintan and
Fumis, Luca and McNeill, Graham and Hayhurst, James D and Martinez Osorio, Ricardo Esteban and
Barkhordari, Ehsan and Ferrer, Javier and Carmona, Miguel and Uniyal, Prashant and Falaguera, Maria J and
Rusina, Polina and Smit, Ines and Schwartzentruber, Jeremy and Alegbe, Tobi and Ho, Vivien W and
Considine, Daniel and Ge, Xiangyu and Szyszkowski, Szymon and Tsepilov, Yakov and Ghoussaini, Maya and
Dunham, Ian and Hulcoop, David G and McDonagh, Ellen M and Ochoa, David},
title = {Open Targets Platform: facilitating therapeutic hypotheses building in drug discovery},
journal = {Nucleic Acids Research},
volume = {53},
number = {D1},
pages = {D1467--D1475},
year = {2025},
doi = {10.1093/nar/gkae1128}
}
- Ensembl: Ensembl places no restrictions on access to or use of its data.
- SCREEN: ENCODE's data-use policy places no restrictions on released data.
- GENCODE: GENCODE states that all its data is open access.
- Open Targets: Open Targets Platform data are CC0 1.0.
- Code: the regions source and the region code are part of
altar, under its MIT licence.
Python and extension¶
RegionAnnotationSource computes the columns in Python, with the same function the annotate command uses.
RegionAnnotationSource.from_options() builds the same source as a run spec, with the same options as keywords:
from altar.sources import fetch_annotations
from altar.variants import RegionAnnotationSource
source = RegionAnnotationSource(data_dir="altar-variants-data")
source.identity() # the contract identity, with the release from region_provenance.json
annotations = await fetch_annotations([source], ["chr1:1014143:C:T"])
from altar.variants import REGION_ANNOTATION_CONTRACT gives the contract. Importing it loads no lookup data.
The annotate command¶
At warehouse scale, run the annotate command, for example in the variants runtime image, and load its JSONL into a table you own:
The input is a headerless TSV of chr, pos, ref, alt and the canonical variant ID. Give every row its ID,
because the table is keyed by variant_id; without it the command writes "variant_id": null and the load is
refused. BigQueryAnnotationSource.export_unannotated writes this input, with the ID, for the variants a table
lacks. --data_dir names the data directory and -b the batch size (default 2,000).
The command writes the source's columns to each JSONL row, and two groups of fields outside the source:
ref_length,alt_length,variant_lengthandvariant_typederive from the alleles alone.in_otand theaf_*fields are population allele frequencies from the Open Targets Platform 25.03 variant index: the gnomAD adjusted frequencies (*_adj) Open Targets publishes for GRCh38 alleles.
| Field | Meaning |
|---|---|
in_ot |
The allele is in the Open Targets variant index |
af_afr, af_ami, af_amr, af_asj, af_eas, af_fin, af_mid, af_nfe, af_sas, af_remaining |
Alternate-allele frequency in that gnomAD population |
A missing frequency is null, not 0.0. An allele not in the index has in_ot false and every af_* null. An
allele in the index keeps null for each population Open Targets does not report, and 0.0 means Open Targets
reports a frequency of zero. Frequencies keep the full double-precision value, so a rare-variant cutoff such as
1e-5 applies to the reported value. For frequencies from a gnomAD release you choose, with counts, grpmax and
filter status, use the gnomAD source.
After the last row, the command writes annotated.jsonl.identity.json, the identity the rows were produced
under:
{
"format": 1,
"identity": {
"source_id": "org.kundajelab.altar.annotation.regions",
"schema_version": "1.0.0",
"release": "ensembl-116.ccre-v4.genes-gencode-47.logic-1",
"genome_build": "hg38"
},
"identity_hash": "…",
"columns": ["region_type", "region_labels", "…"],
"provenance": {"format": 2, "genome_build": "hg38", "logic_version": 1, "inputs": {"…": "…"}, "derived": {"…": "…"}},
"row_count": 2
}
A data directory without a provenance manifest still annotates, with a warning and no sidecar. A run removes any sidecar an earlier run left at the same path.
Load the output into a table¶
The BigQuery backend is Preview: tested on the BigQuery emulator and DuckDB; not run against live BigQuery. Declare the table with the contract's columns and the release it serves, then load the JSONL with the sidecar's identity. Altar does not read object storage itself, so read the sidecar's bytes with your own storage client:
from altar.sources import BigQueryAnnotationSource, load_annotation_identity
from altar.variants import REGION_ANNOTATION_CONTRACT
regions = BigQueryAnnotationSource.connect(
"my-project.annotations.variant_regions",
REGION_ANNOTATION_CONTRACT.columns,
name="regions",
genome_default="GRCh38",
identity=REGION_ANNOTATION_CONTRACT.identity(
release="ensembl-116.ccre-v4.genes-gencode-47.logic-1", genome_build="hg38"
),
identity_scoped=True,
)
await regions.ensure_schema()
sidecar_bytes = my_storage_client.read_bytes("gs://my-bucket/run-7/annotated.jsonl.identity.json")
await regions.add_annotations_from_uri(
"gs://my-bucket/run-7/*.jsonl", expected_identity=load_annotation_identity(sidecar_bytes)
)
A sidecar that names another identity than the table's is refused before anything is written. The loader checks
each row's chr, pos, ref and alt against its variant_id, ignores the fields outside the contract, and
skips variants the table already holds under this identity. SqliteAnnotationSource takes the same columns,
identity and identity_scoped.
The identity, not the source class, satisfies ChromBPNet's dependency, so a table declared this way and
RegionAnnotationSource are interchangeable. A table built earlier by equivalent logic from the same inputs can
declare the release above without a sidecar; declare it only when that is true. See
Serve annotations from your own tables for identity scoping and loading, and
Run at warehouse scale on BigQuery.