gnomAD¶
The gnomad source adds how common each variant is in gnomAD v4: allele
frequencies overall and per genetic ancestry group, allele counts, and the site's filters. It is a lookup in a
released dataset and runs no model.
| Kind | Annotation source; type = "gnomad" |
| Install | pip install altar-popfreq (PyPI) |
| Genome builds | hg38 |
| Variant classes | SNVs and indels, on chr1–22, X and Y |
| Data | Published: release 4.1.1-genomes on hg38, read in place. gcs (default): gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes; hf: hf://datasets/altar-genomics/gnomad-4.1.1-genomes. Copy it with altar pull gnomad --to PATH_OR_URI, or --bigquery PROJECT.DATASET.TABLE. Build your own with altar-popfreq-build |
| Validation | Matches upstream on spot checks (Validation) |
| Licence | CC0 data; cite gnomAD |
| Upstream | Data, Paper, Documentation |
When to use it¶
- Use gnomAD to ask how common a variant is in the general population, overall and in each genetic ancestry group, and to filter a cohort down to rare variants (Filtering).
- A variant gnomAD has no row for gets no columns. An absent frequency is unknown, not zero.
- Choose ClinVar to ask whether submitters have classified a variant as pathogenic or benign. gnomAD holds frequencies and no clinical assertions. Rare-disease filtering often reads both.
- Choose a model, such as Cherimoya, or a precomputed prediction, such as AlphaMissense, to predict what a variant does. A frequency says nothing about effect.
- gnomAD serves hg38 only and has no chrM rows here (Limits).
Add it to a run¶
- Install the binding:
pip install altar-popfreq. - Add the section above to the run spec. It reads the published copy in place, with no download and no credentials.
altar sources gnomad lists the columns, the options and the published copy. The
run spec reference describes the [sources.<name>] section, and
Annotate variants with evidence walks through a run.
Tutorial 0: look up what is known about five variants looks five
variants up in gnomAD in a notebook.
Outputs¶
| Column | Type | Meaning |
|---|---|---|
gnomad_af |
float | Alternate allele frequency, AC / AN. Null when AN is 0. |
gnomad_ac |
int | Alternate allele count. |
gnomad_an |
int | Number of called alleles. |
gnomad_nhomalt |
int | Number of homozygous-alternate individuals. |
gnomad_af_grpmax |
float | gnomAD's grpmax frequency: the greatest frequency over the groups gnomAD includes in grpmax. |
gnomad_grpmax |
str | The group that frequency comes from, such as nfe. |
gnomad_faf95_max |
float | gnomAD's maximum filtering allele frequency: the greatest per-group FAF (Poisson 95% CI) over the grpmax groups. Null when no such group has more than one alternate allele. |
gnomad_af_max_ancestry |
float | The greatest non-null frequency over the ten ancestry columns. Null if all ten are null. |
gnomad_af_*, 10 columns: afr, ami, amr, asj, eas, fin, mid, nfe, sas, remaining |
float | Frequency within each genetic ancestry group. |
gnomad_filters |
str | The site's VCF FILTER value: PASS, or the ;-joined filters it failed, such as AC0;AS_VQSR. |
Every column starts with gnomad_.
gnomAD leaves its bottlenecked groups out of grpmax and the filtering allele frequency. Which groups it leaves out depends on the dataset:
| Dataset | Groups grpmax and the FAF cover | Left out |
|---|---|---|
| genomes | afr, amr, eas, nfe, sas |
ami, asj, fin, mid, remaining |
| exomes | afr, amr, eas, mid, nfe, sas |
asj, fin, remaining (exomes have no ami) |
| joint | afr, amr, eas, mid, nfe, sas |
ami, asj, fin, remaining |
gnomad_af_max_ancestry covers all ten groups in every dataset. A column the release does not publish is null:
the v4 exomes VCFs have no AF_ami, so gnomad_af_ami is null in an exomes build.
A variant gnomAD has no row for gets no gnomAD columns. It is not given a frequency of zero, because an unknown frequency is not evidence that a variant is rare. Filtering shows how to decide what an absent variant means.
An augmented VCF written from a run's export carries these columns as ALTAR1_GNOMAD_* INFO fields; see
Write an augmented VCF.
Configuration¶
| Option | Set in | Type | Default | Meaning |
|---|---|---|---|---|
dataset |
[sources.gnomad.options] |
path | not set | your own copy, as a path or URI, such as one altar pull made (default: the published copy) |
table |
[sources.gnomad.options] |
text | not set | your own BigQuery table, project.dataset.table, such as one altar pull --bigquery loaded |
release |
[sources.gnomad.options] |
text | not set | the release a table holds, given only with table (default: the published release) |
genome_build |
[sources.gnomad.options] |
text | not set | the genome build of the data, checked against it when given |
storage_options |
[sources.gnomad.options] |
table | not set | fsspec settings for reading the files, such as credentials |
connection |
[sources.gnomad] |
not set | connection = "<name>": the [connections.<name>] section to read the data through | |
location |
[sources.gnomad.options] |
one of "gcs", "hf" |
"gcs" |
where to read the published copy from |
Every option is optional. With none, the source reads the published copy from gcs.
location = "hf"reads the Hugging Face mirror, which needspip install "altar[hf]"(Where it comes from).datasetnames your own Parquet copy, as a path or a URI.tablenames your own BigQuery table,project.dataset.table, read through agcpconnection. It is Preview: tested on the BigQuery emulator and DuckDB; not run against live BigQuery.releaseis given withtableonly, such as4.1-joint. The default is the published release.genome_build, when given, must behg38.storage_optionsis not allowed withconnection.connection = "<name>", set in[sources.gnomad]itself, names the[connections.<name>]section to read through.
An unknown option stops the run with an error that lists the valid ones.
Inputs¶
- Genome builds: hg38 only. A run on another build stops before it reads the cohort, and the error names the
source. Only gnomAD v4 on GRCh38, which Altar calls
hg38, is accepted. gnomAD v2.1.1 is on GRCh37 but uses other INFO names and ancestry groups (popmaxinstead ofgrpmax,othinstead ofremaining, noamiormid), so its values would not mean the same thing under these columns. - Variant classes: the SNVs and indels gnomAD holds, on chr1–22, X and Y. The published copy has no chrM rows.
- Allele matching: gnomAD's keys are split, trimmed and left-aligned. Ingestion with a reference FASTA gives a cohort the same keys. A lookup takes variant IDs as given. An indel that was only trimmed, not left-aligned, finds no row. See Variant identity and normalization.
Where it comes from¶
The Kundaje lab builds the latest release, gnomAD v4.1.1 genomes on hg38, with altar-popfreq-build. It
publishes the output in a public-read bucket, with a byte-for-byte mirror on Hugging Face
(dataset card). The card above lists both
locations. It holds 759,336,320 alleles in 24 files, 24.9 GiB. altar-popfreq depends on altar[gcs], so the
default location needs no other install.
location = "hf" reads the mirror, which needs pip install "altar[hf]". Hugging Face limits anonymous
requests, so a large cohort read from hf can stall; see
Hugging Face rate limits.
To keep your own copy:
altar pull gnomad --to gs://my-bucket/gnomad-4.1.1-genomes
altar pull gnomad --bigquery my-project.reference.gnomad_4_1_1_genomes --project my-project
The first copies the Parquet files and checks that every file records the release. The second loads them into a
BigQuery table clustered on variant_id and labels the table with its release. altar pull --bigquery is Preview:
tested against a fake BigQuery client only; not run against live BigQuery. Name the copy with dataset or table. A
copy in your bucket is read through a connection:
[connections.lab]
type = "gcp"
options = { project = "my-project" }
[sources.gnomad]
type = "gnomad"
options = { dataset = "gs://my-bucket/gnomad-4.1.1-genomes" }
connection = "lab"
Choose where data is read from covers the published copy, your own copy, BigQuery
tables, altar pull and connections for every source.
Release and identity¶
A gnomAD release is named by gnomAD's version, exactly as it appears in the file names, and by the dataset.
gnomad.genomes.v4.1.1.sites.chr1.vcf.bgz is version 4.1.1 of the genomes dataset. Use 4.1.1, not 4.1:
a patch release changes values. The dataset is genomes, exomes or joint.
The release label is <version>-<dataset>, such as 4.1.1-genomes. Frequencies from different datasets or
versions are different quantities, so each label is a different identity.
The source's identity is the contract org.kundajelab.altar.annotation.gnomad (version 1.0.0), the release
label and the genome build. Where the data is read from is not part of it. The published copy, a pulled copy and
a BigQuery table of one release declare the same identity, so switching between them keeps every stored
annotation reusable.
Build another release¶
Build a release yourself to use the exomes or joint dataset, or a v4 release the lab does not publish.
altar-popfreq-build streams gnomAD's sites VCFs into Parquet, one row per allele, keyed by the canonical
chr:pos:ref:alt:
altar-popfreq-build --dataset genomes --release 4.1.1 --workers 8 \
--input gnomad.genomes.v4.1.1.sites.chr*.vcf.bgz \
--output ./gnomad-4.1.1-genomes
--input and --output also take URIs (gs://, s3://, https://, hf://). This builds chr21 from
gnomAD's public bucket into yours, with no download step:
altar-popfreq-build --dataset genomes --release 4.1.1 \
--input gs://gcp-public-data--gnomad/release/4.1.1/vcf/genomes/gnomad.genomes.v4.1.1.sites.chr21.vcf.bgz \
--output gs://my-bucket/gnomad-4.1.1-genomes/
A scheme other than gs:// needs its provider's Altar extra, such as altar[http] for https://.
--workers builds that many inputs at once, in separate processes; give one input per chromosome to use several
cores. --input - reads one VCF from standard input, and --stdin-name names it.
The builder:
- writes
<output>/chr=<chromosome>/<input name>.parquet, with the release, dataset, genome build and builder version in each file's metadata; - reads the dataset's INFO names: the joint dataset's carry a
_jointsuffix, such asAF_joint; - checks inputs named like gnomAD's files against
--datasetand--release, and rejects a header whosechr1length is not GRCh38's; - splits multi-allelic records, trims each allele, writes
chrcontig names, and skips symbolic,*and non-ACGT alleles, which have no canonical key; - rejects input that is not sorted by position, and a variant key that occurs twice across all inputs;
- rejects a bgzipped input that does not end with bgzip's end-of-file block. Plain text or plain gzip has no such marker, so stream bgzipped files;
- writes only into an empty or absent
--output.--overwritedeletes a previous build there first.
A URI output is resumable. Each input's files are uploaded as it finishes, so running the same command again
after a crash skips the inputs already written. A remote input whose stream fails is read again from the start,
up to --input-attempts times. --input-storage-option and --output-storage-option pass non-secret fsspec
options, one KEY=VALUE each. pip install "altar-popfreq[build]" names the builder's dependencies, which
altar already installs.
Name the output with dataset. The source reads the release from the files' metadata and refuses a dataset
whose files record no release.
Compute and scale¶
- A lookup runs no model and needs no GPU.
- The published copy is 759,336,320 alleles in 24 files, 24.9 GiB. A lookup reads only the chromosome partitions and row groups its variants fall in, not the whole release.
- Hugging Face limits anonymous requests, so read a large cohort from
gcsor from your own copy. - Most of a cohort's variants are in gnomAD. On BigQuery, read a loaded table in the store's own SQL with
altar_popfreq.bigquery_sourcerather than staging Python lookups (Python and extension).
Prioritization¶
None. A frequency says how common a variant is, not that it is interesting, so the source never sets
prioritized. Use the columns as a filter (Filtering).
What prioritized means explains how rules combine.
Filtering¶
For rare-disease filtering, gnomAD recommends the
filtering allele frequency, gnomad_faf95_max, compared against
the maximum credible allele frequency of the disease. A rarity rule that also keeps the variants gnomAD lacks,
written with predicates:
from altar.predicates import Col, IsNull, Le, Lit, Or
rare = Or(IsNull(Col("gnomad_faf95_max")), Le(Col("gnomad_faf95_max"), Lit(1e-4)))
rare.to_sql() # ((gnomad_faf95_max IS NULL) OR gnomad_faf95_max <= 0.0001)
rare.to_python() returns the same rule as a function of a row of the export.
What the FAF is, in gnomAD v4:
- For each group, it is a lower bound on the group's frequency from a Poisson 95% confidence interval. A few alleles in a small group do not make a variant look common.
gnomad_faf95_maxis the greatest per-group FAF. Its group need not be thegnomad_grpmaxgroup.- It covers only the grpmax groups in the table under Outputs, never
ami,asj,finorremaining. - A group with at most one alternate allele has a FAF of 0, and gnomAD reports the maximum only over groups above
- The FAF is therefore null for a singleton, for a variant seen only in left-out groups, and for an
AC0site.
A founder variant common in Ashkenazi Jewish or Finnish individuals, and rare in every covered group, passes a
FAF filter. For a cohort drawn from a bottlenecked group, or to drop a variant common in any group, add a rule on
gnomad_af_max_ancestry:
from altar.predicates import And, Col, IsNull, Le, Lit, Or
rare_everywhere = And(
Or(IsNull(Col("gnomad_faf95_max")), Le(Col("gnomad_faf95_max"), Lit(1e-4))),
Or(IsNull(Col("gnomad_af_max_ancestry")), Le(Col("gnomad_af_max_ancestry"), Lit(1e-3))),
)
gnomad_af_max_ancestry is a point estimate with no confidence bound. One or two alleles in a small group, such
as ami, give a high value, so give it a looser threshold than the FAF.
In both rules, null passes. Le(...) alone drops the variants gnomAD lacks. IsNull(Col("gnomad_ac")) tells an
absent variant from a present one with a null FAF. An AC0 site has gnomad_af 0.0 and a null FAF, so a rarity
rule counts it as observed and rare. Add Eq(Col("gnomad_filters"), Lit("PASS")) or
Gt(Col("gnomad_ac"), Lit(0)) if you mean an observed, passing frequency.
Validation¶
Validation label: Matches upstream on spot checks. When the published copy was built, the alleles written plus the records skipped were checked against the number of upstream VCF records the builder read. APOE rs429358 and rs7412, CFTR F508del and BRCA1 185delAG were looked up in the published copy and in gnomAD, and agreed.
Not validated:
- a release or dataset you build yourself, such as the exomes or joint dataset;
- reading a BigQuery table (
table) or loading one (altar pull --bigquery) on live BigQuery, both Preview.
Limits¶
- The published v4.1.1 genomes data covers chr1–22, X and Y. gnomAD publishes mitochondrial variants separately, in v3.1 with other fields, which this source does not read. Every chrM variant has no gnomAD columns, so a rarity rule in which null passes keeps it. Treat a chrM variant's frequency as unknown.
- An indel that was only trimmed, not left-aligned, finds no row, and a rarity rule in which null passes keeps it however common it is (Inputs).
- hg38 only. A run on another build stops before it reads the cohort, and the error names the source.
Cite and licence¶
gnomAD's citation policy asks you to cite its flagship paper, which describes the gnomAD v3 genomes, for data from any release:
@article{chen2024gnomad,
author = {Chen, Siwei and Francioli, Laurent C. and Goodrich, Julia K. and Collins, Ryan L. and Kanai, Masahiro
and Wang, Qingbo and Alf{\"o}ldi, Jessica and Watts, Nicholas A. and Vittal, Christopher and
Gauthier, Laura D. and Poterba, Timothy and Wilson, Michael W. and Tarasova, Yekaterina and Phu, William and
Grant, Riley and Yohannes, Mary T. and Koenig, Zan and Farjoun, Yossi and Banks, Eric and Donnelly, Stacey
and Gabriel, Stacey and Gupta, Namrata and Ferriera, Steven and Tolonen, Charlotte and Novod, Sam and
Bergelson, Louis and Roazen, David and Ruano-Rubio, Valentin and Covarrubias, Miguel and
Llanwarne, Christopher and Petrillo, Nikelle and Wade, Gordon and Jeandet, Thibault and Munshi, Ruchi and
Tibbetts, Kathleen and {Genome Aggregation Database Consortium} and O'Donnell-Luria, Anne and
Solomonson, Matthew and Seed, Cotton and Martin, Alicia R. and Talkowski, Michael E. and Rehm, Heidi L. and
Daly, Mark J. and Tiao, Grace and Neale, Benjamin M. and MacArthur, Daniel G. and Karczewski, Konrad J.},
title = {A genomic mutational constraint map using variation in 76,156 human genomes},
journal = {Nature},
volume = {625},
number = {7993},
pages = {92--100},
year = {2024},
doi = {10.1038/s41586-023-06045-0}
}
Name the release in gnomAD's form, such as "gnomAD (v4.1, 807,162 individuals, released on 04/19/2024) (Chen et al., 2023)", and link to the browser from an online resource.
- Data: gnomAD's terms of use release its primary data under CC0. gnomAD asks users to cite it and name the release, not to attempt to re-identify participants, and to contact the consortium before large-scale analyses that may overlap ongoing work. The builder reads only the frequency, count and filter fields above. Other annotations in gnomAD's VCFs carry other licences, and the published copy holds none of them.
- Binding:
altar-popfreqis MIT-licensed. It ships no gnomAD data. Its name describes the data rather than naming gnomAD, as gnomAD's policies ask. gnomAD is a trademark of The Broad Institute, Inc.; the binding is not affiliated with or endorsed by the gnomAD project.
Python and extension¶
GnomADSource.from_options() builds the same source as a run spec, with the same options as keywords.
Look up evidence shows a lookup in several sources at once.
GnomADSource.published_dataset() returns the published locations and identity.
On BigQuery, read a loaded table in the store's own SQL with altar_popfreq.bigquery_source rather than staging
Python lookups. Run at warehouse scale on BigQuery covers it, and what each
read path checks.