Skip to content

Annotate variants with evidence

An annotation run looks each variant up in evidence datasets, such as gnomAD allele frequencies and ClinVar classifications, and writes one table with a row per variant. It runs no model, so it needs no Docker, GPU or reference genome.

Before you start

  • Start 1 of the quickstart: its environment, cohort.tsv and annotate.toml. The examples add to that spec.
  • Network access. gnomAD, ClinVar, Open Targets E2G and phyloP are read from a public Google Cloud bucket, anonymously, and a lookup reads only the parts of each dataset its variants fall in.
  • Optional: a reference FASTA of the cohort's build, to left-align indels. See Variants a lookup may miss.

Install the bindings of the sources you add. The examples add phyloP and Open Targets E2G:

pip install altar-conservation altar-opentargets-e2g

Add a source to the spec

altar sources lists the installed annotation sources, with the name a spec gives as the source's type (abbreviated):

$ altar sources
Annotation sources add columns to your variants. To use one, put its name in a run spec as the
source's type:

  [sources.clinvar]
  type = "clinvar"

Run `altar sources <name>` to see a source's columns, options and data.

Ready-to-use sources
  clinvar (altar-clinvar)
      ClinVar's germline classification of each variant, with its review status, conditions and
      genes.
      type = "clinvar"; 18 columns; published release 20260928 (hg38), read in place
  gnomad (altar-popfreq)
      How common each variant is in gnomAD v4: allele frequencies overall and per genetic ancestry
      group.
      type = "gnomad"; 19 columns; published release 4.1.1-genomes (hg38), read in place
  ...

altar sources gnomad shows one source in full: its columns, a section to paste, its options and where its data is published.

Each [sources.<name>] section adds one source. Add phyloP conservation and Open Targets E2G enhancer-to-gene links to the quickstart's annotate.toml:

[sources.phylop]
type = "phylop100way"

[sources.e2g]
type = "opentargets_e2g"

A source's settings go in its options. ClinVar, for example, flags reviewed pathogenic variants as prioritized when asked. Change its section to:

[sources.clinvar]
type = "clinvar"
options = { prioritize_pathogenic = true }

gnomAD, ClinVar, phyloP and Open Targets E2G read a published copy of their data by default. AlphaMissense, SpliceAI, GPN-Star, the E2G atlases and genomic regions read data you build or download, which the section names with an option. Each source's page has a section to start from and lists its options, and the catalog lists every source.

To read a published dataset from the Hugging Face mirror, your own copy or a BigQuery table, see Choose where data is read from. The run spec reference lists the keys of a source section.

Run it

Check the spec with altar run --validate annotate.toml (Provisional). It opens each source and says what the run would do, without reading the cohort; Check the result shows its output. Then run it:

$ altar run annotate.toml
[sources.gnomad] opened gnomad 4.1.1-genomes (hg38) from gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes (published, gcs) in 1.4s
[sources.clinvar] opened clinvar 20260928 (hg38) from gs://kundajelab-altar/clinvar/clinvar-20260928-hg38 (published, gcs) in 0.3s
[sources.phylop] opened phylop100way 2015-05 (hg38) from gs://kundajelab-altar/conservation/phylop100way-hg38 (published, gcs) in 0.4s
[sources.e2g] opened opentargets_e2g 26.09 (hg38) from gs://kundajelab-altar/opentargets-e2g/enhancer-to-gene-26.09 (published, gcs) in 0.4s
[variants] reading cohort.tsv (trimming only, no reference genome)
[variants] read 5 variants in 0.0s
[export] annotating 5 variants with 4 sources; writing parquet to first-annotation/export
[export] wrote 5 rows to first-annotation/export in 1.8s; _columns.tsv lists its columns
done in 5.9s
altar run: warning: annotation sources may miss 1 indel: chr7:117559590:ATCT:A
  It was trimmed but not left-aligned, because [variants] names no reference genome; gnomAD and ClinVar store indels left-aligned.
  Set [variants] reference or reference_fasta to left-align it. Every variant is still in the export.

Each source's line names the release and genome build it opened. Progress goes to standard error, and the run summary is printed as JSON on standard output. A spec without models keeps no store file and creates no cache: the run assembles the table in memory and writes it to <work_dir>/export, first-annotation/export here, in a directory named after job_id. Choose where results are stored covers the stores a scoring run keeps.

Variants a lookup may miss

Every variant of the cohort is in the export. A source may still fail to find a variant it holds, or look it up as another variant. The run counts these variants and ends with a warning for each kind: what it is, with an example, why, and what to do, as the end of the run's output shows. A warning does not fail the run. The summary's potential_misses lists the same kinds, each with its reason, count, example and message.

The four kinds, by reason:

  • indel_not_left_aligned: the spec has no reference_fasta, so indels are only trimmed. gnomAD and ClinVar key indels at their leftmost position in a repeat, so an indel written further right is missed. Every indel is counted, since none could be checked.
  • reference_mismatch: REF is not the reference's bases at that position. The message shows what the reference holds. The cohort may be on another genome build.
  • contig_not_in_reference: the FASTA has no contig of that name, under any spelling.
  • contig_not_primary: the variant is on a contig other than chr1–22, X, Y and M, such as an alt contig. The published datasets cover the primary chromosomes only, so its source columns are empty.

Fix a run that stops or warns says what to do for each.

In the quickstart cohort, the ΔF508 deletion chr7:117559590:ATCT:A is already written at its leftmost position, so gnomAD and ClinVar find it. An indel written further right in a repeat would be missed without a reference.

Check the result

Read the new columns from the export:

import pandas as pd

table = pd.read_parquet("first-annotation/export")
columns = ["variant_id", "prioritized", "clinvar_classification", "phylop100way", "link_top_gene", "link_gene_count"]
print(table[columns].to_string(index=False))
           variant_id  prioritized       clinvar_classification  phylop100way   link_top_gene  link_gene_count
   chr17:43045712:T:C         True Pathogenic/Likely_pathogenic         4.245             NaN              NaN
   chr6:136273879:G:A        False                          NaN        -0.788             NaN              NaN
   chr6:136274004:C:T        False                          NaN         5.752             NaN              NaN
   chr6:136274285:A:G        False                          NaN         1.942             NaN              NaN
chr7:117559590:ATCT:A         True                   Pathogenic           NaN ENSG00000077063              2.0

The table now has 48 columns: the quickstart's 41, phyloP's phylop100way, and six Open Targets E2G link columns. Each source's page lists its columns and what each means.

  • prioritized is true for the two variants ClinVar classifies as pathogenic, because the spec sets prioritize_pathogenic. prioritized_source_names holds ["clinvar"] for both.
  • phyloP scores SNVs by default, so the deletion's phylop100way is empty.
  • Open Targets has two links for the deletion, from regulatory elements in two biosamples to two genes, one of them CFTR (ENSG00000001626). link_top_gene is the gene of the stronger link, and link_genes holds both. The SNVs fall in no linked element, so their link columns are empty.

An empty value means the source holds no record for that variant. It is not a zero. What "prioritized" means describes how a missing value enters a prioritization rule.

The export directory also holds _run_summary.json, which names the release and genome build each source served. Read a run's results describes the export and its provenance.

If it fails

  • A misspelled type stops the run with exit status 2, and the message suggests the installed name, such as did you mean 'gnomad'?. altar sources lists the names.
  • A source that cannot serve the run's genome build stops the run before it reads the cohort; see Match the genome build.

Fix a run that stops or warns covers the other messages, including a missing extra and a lost connection.

Next