Skip to content

Ingest variants and write augmented VCFs

altar.variants ingests a VCF or a variant TSV against a reference genome. It writes every ALT allele with its record number, the canonical variants to score, and a manifest. After scoring, it writes the results back into the same VCF as INFO fields, the augmented VCF. Run these steps yourself when your own application keeps a cohort's records, samples and genotypes beside its scores. altar run reads a VCF itself; for that path see Give a run its variants.

The ingest and export steps are commands, run with python -m. The functions altar.variants exports are stable, except the names Provisional names lists. Import from altar.variants itself, not from the modules below it.

Run the ingest pipeline

The pipeline reads a coordinate-sorted VCF, a BGZF-compressed VCF, or a headerless TSV with the columns chr, pos, ref, alt and an optional fifth ID column. It reads the input through htslib (pysam) a batch of ALT alleles at a time, so a large callset never sits in memory.

python -m altar.variants.preprocessing.pipeline \
    -i input.vcf.gz \
    -g genome.fa \
    --genome-build hg38 \
    -o valid.tsv \
    --invalid-out invalid.tsv \
    --region-out-dir regions/ \
    --occurrence-out-dir ingest/occurrences/ \
    --canonical-out-dir ingest/canonical/valid/ \
    --invalid-parquet-out-dir ingest/canonical/invalid/ \
    --manifest-out ingest/manifest.json \
    --header-out ingest/header.vcf

The last five options are optional. Without them, the pipeline writes the same files under an ingest/ directory beside valid.tsv. -f sets the input format (auto, tsv or vcf); auto, the default, reads the file extension.

--genome-build names the assembly of the input and of the FASTA, such as hg38 or mm10. The default is hg38. A GRC name is recorded in Altar's spelling, so GRCm38 becomes mm10, and an unknown assembly is rejected before the input is read. The build becomes the genome of every ingested row and the manifest's reference_build. Score the cohort with run inputs on the same build. The region-category TSVs come from hg38 annotation data, so on any other assembly they are empty.

For a multi-sample VCF where only some samples are cases, pass --target-samples-file with one VCF sample ID per line. An ALT is kept for scoring when at least one selected sample carries it. The manifest records a digest and a count of the selected samples, not their IDs.

Ingest a callset split across files

For a callset split into several VCF files, ingest each file with selected_vcf_ingest. It checks the file and the sample list against digests you recorded, and skips region filtering:

python -m altar.variants.preprocessing.selected_vcf_ingest \
    --input shard-01.vcf.gz --genome genome.fa --genome-build hg38 \
    --selected-samples-file case-samples.txt --output-dir shard-01-ingest \
    --expected-input-sha256 "$(sha256sum shard-01.vcf.gz | cut -d' ' -f1)" \
    --expected-selected-samples-sha256 "$SAMPLES_SHA256"

The sample digest is the hex SHA-256 of the sorted sample IDs as a JSON list. altar.variants.canonical_sample_selection_digest(samples) computes it (provisional). Once every file's ingest has written its manifest, merge the canonical variants and drop repeats:

python -m altar.variants.preprocessing.callset_union \
    --ingest-dir shard-01-ingest --ingest-dir shard-02-ingest \
    --output-dir callset-union

The union rejects an ingest whose outputs changed, or that ran on another build or under another normalization policy. It writes its own manifest last. Neither command decides which samples are cases; you choose them, and the sample digest records the choice.

What the pipeline writes

The pipeline writes:

Output Contents
valid.tsv Headerless chr, pos, ref, alt, source_variant_id rows, one per scoreable allele.
invalid.tsv One row per rejected allele, with a stable error code.
regions/ One TSV per region category, such as coding or promoter.
ingest/occurrences/ Parquet files with one row per ALT of every record, including unsupported alleles, keyed by (record_ordinal, alt_index).
ingest/canonical/valid/ Parquet files of the scoreable alleles, by canonical variant ID.
ingest/canonical/invalid/ Parquet files of the rejected alleles.
ingest/header.vcf The input's parsed header.
ingest/manifest.json Input and header digests, counts, each Parquet file's row count and digest, the reference build, and the canonicalizer, validation and normalization versions.

Each Parquet file is renamed into place once complete, beside a success marker with its row count, size and SHA-256 digest. The manifest is written last and marks the whole ingest complete. The Parquet schemas are OCCURRENCE_ARROW_SCHEMA, CANONICAL_ARROW_SCHEMA and INVALID_ARROW_SCHEMA in altar.variants (provisional).

How records become rows:

  • A multi-allelic record gives one occurrence per ALT, in the record's order.
  • A repeated record gives repeated occurrences. The canonical ID is chr:pos:ref:alt, and a consumer selects it distinctly.
  • The VCF ID, or the fifth TSV column, becomes source_variant_id; see Custom variant IDs.
  • A spanning deletion *, a symbolic structural variant, a breakend, or an ALT equal to REF stays in the occurrences with status UNSUPPORTED and an error code. It is not sent to scoring.
  • The chromosome name is normalized during reference validation. POS stays one-based.
  • An allele that passes the reference check is trimmed and left-aligned before its ID is assigned. The canonical shards and valid.tsv carry the normalized locus, and the occurrence keeps the input's spelling in its original_* columns. The manifest records the policy as normalization_policy, altar-normalize-v1.
  • INFO, FORMAT and sample columns are not copied. The input VCF keeps them for the augmented VCF.

Variant identity and normalization explains the reference checks and the normalization rule.

Inputs the pipeline refuses

The pipeline refuses these inputs before it writes any output:

  • a gVCF: a header declaring ##GVCFBlock* or ##ALT=<ID=NON_REF,...>, or, without a declaration, the first <NON_REF>, <*> or END > POS reference block without a concrete ALT. A symbolic structural variant with END and a spanning deletion * are not taken for gVCF records.
  • unsorted VCF records, because the augmented VCF carries a CSI index;
  • BCF, with a message to convert it to VCF first.

Load a small input as a DataFrame

The DataFrame loaders, read_variants_frame, load_variants and load_variants_vcf, return a whole pandas DataFrame for a small input. Their variant_id is the canonical ID, and they do not keep the VCF ID. They do not refuse a gVCF: for VCF input they drop and count symbolic ALTs such as <NON_REF> and any ALT with no variant key, and keep concrete alleles. A TSV has no such records, so load_variants and read_variants_frame reject a TSV at the first row with no key and name the row. Every loader rejects a position that is not ASCII digits. They only trim alleles; validate_variants(frame, genome) checks each row against the reference and left-aligns the valid ones. Use the pipeline for a large file.

Build and check variant IDs

A variant ID is chr:pos:REF:ALT, such as chr1:100:A:G. canonical_variant_id(chromosome, position, ref, alt) builds one from locus fields, and VariantKey parses and checks one:

from altar.variants import VariantKey, canonical_variant_id

canonical_variant_id("1", 100, "a", "g")  # 'chr1:100:A:G'
key = VariantKey.require_canonical("chr1:100:A:G")
key.chromosome, key.position, key.reference_allele, key.alternate_allele  # ('chr1', 100, 'A', 'G')

Both are also exported from altar.models and altar.sources, and come from the altar-identity package that model runtimes import, so every runtime builds the same IDs. Scoring and assembly reject an ID that is not canonical.

An indel can be written in more than one place. minimal_representation(position, ref, alt) trims the bases REF and ALT share. left_align(chromosome, position, ref, alt, genome) trims and then shifts an indel left through a repeat, with genome an open pyfaidx.Fasta of the cohort's build. An annotation lookup trims an ID but never left-aligns it, so an indel written at the right end of a repeat finds no gnomAD or ClinVar row. Left-align IDs that did not come from the pipeline or from altar run with a reference before you look them up:

import pyfaidx
from altar.variants import canonical_variant_id, left_align

genome = pyfaidx.Fasta("hg38.fa")
position, ref, alt = left_align("chr1", 107, "AA", "A", genome)
variant_id = canonical_variant_id("chr1", position, ref, alt)

validate_variant(chromosome, position, ref, alt, genome) checks one variant against the reference, and validate_variants(frame, genome) checks a DataFrame and returns the valid and invalid rows.

Custom variant IDs

The VCF ID and the optional fifth TSV column are labels from the input, such as rs123 or a cohort's own IDs. They are not join keys. The pipeline keeps them as source_variant_id in valid.tsv and the occurrences, where the augmented VCF can use them. Scores, details and annotations join on the canonical ID, which model runtimes always derive from chr, pos, ref and alt. The DataFrame loaders do the same and drop the input's IDs; use the pipeline's outputs when you need them.

The genome build is not part of the ID. It travels with source identities and run provenance, so the same coordinates on two assemblies are never taken for one variant.

Write an augmented VCF

The augmented VCF is the input VCF with Altar's results added as INFO fields, aligned with each record's ALT order. It is written from two inputs: the unchanged input VCF, and occurrence buckets that join each ALT to its result row.

  • BigQueryScoreStore.export_occurrence_buckets joins the ingest's occurrence table to the results in SQL and writes the buckets.
  • altar.results.write_occurrence_buckets writes the same buckets from any store's assembled results (materialize) and the ingest's occurrence files.
  • After altar run, python -m altar.variants.export.run_export and altar.variants.augment_run_export build the buckets from the run's export and write the VCF in one step; see Write an augmented VCF.

Then write the VCF:

python -m altar.variants.export.augmented_vcf \
    --input-vcf input.vcf.gz \
    --bucket-dir buckets/ \
    --models-json models.json \
    --ingest-manifest ingest/manifest.json \
    --output-vcf cohort.altar.vcf.gz \
    --output-manifest cohort.altar.manifest.json \
    --results-parquet-uri gs://bucket/job/results

It streams the input VCF through htslib and writes a BGZF VCF, its CSI index, and a manifest with checksums. --results-parquet-uri is recorded in the manifest as the results the VCF came from. --annotation-fields-json picks the annotation INFO fields; see Choose the INFO fields.

The writer names no model. ALTAR1_MAX_ABS_LOGFC reads each model's logfc field, a score several models share. --include-model-scores adds one ALTAR1_MODEL_SCORE entry per model, holding the fields models.json lists:

{
  "score_fields": ["logfc", "jsd", "active_allele_quantile", "in_peak"],
  "models": [{"model_id": "...", "name": "...", "column_prefix": "..."}]
}

Use the model manifest's result-schema field names. Each entry is ALT|model|<score_fields...>|prioritized, and the INFO header records that order. --include-model-scores fails without score_fields. column_prefix is the model's prefix in the wide export, altar.results.export_column_prefix(model_id, model_name).

Occurrence buckets

The bucket directory follows the altar-augmentation-buckets-v1 schema, which the output manifest records as bucket_schema_version. It holds bucket=<n>/part-*.parquet files with one row per input ALT. The writer reads these columns, plus the annotation columns its INFO fields name, and ignores the rest:

Column Type Fills From
record_ordinal int alignment the occurrences (zero-based record number)
alt_index int alignment the occurrences (one-based ALT number)
status, error_code str ALTAR1_STATUS, ALTAR1_ERROR the occurrences
result_variant_id str whether a result joined the joined result's ID, null when none joined
prioritized bool ALTAR1_PRIORITIZED the assembled result
<column_prefix>_logfc float ALTAR1_MAX_ABS_LOGFC each model's logfc field
<column_prefix>_<score_field>, <column_prefix>_prioritized any ALTAR1_MODEL_SCORE with --include-model-scores
most_active_celltype str ALTAR1_TOP_MODEL an optional display column naming the headline model

The writer reads the result columns only when result_variant_id is set. PENDING status becomes SCORED when a result joined and NO_RESULT otherwise. A bucket keeps every annotation column of every joined source with its native type, so a column from your own source reaches the writer unchanged.

Choose the INFO fields

The core fields ALTAR1_STATUS, ALTAR1_ERROR, ALTAR1_PRIORITIZED, ALTAR1_MAX_ABS_LOGFC, ALTAR1_TOP_MODEL and ALTAR1_MODEL_SCORE are fixed. Annotation fields are declared: each declaration maps one or more bucket columns to one INFO field.

Declare annotation fields

A declaration has these keys:

Key Required Meaning
id yes The INFO ID ([A-Za-z_][A-Za-z0-9_.]*). Not a core field ID, END, or another declaration's ID.
columns yes Bucket columns tried in order; the first value that is not null or empty after extraction wins.
type yes Integer, Float or String. A boolean writes 1 or 0. A value that does not convert, or an Integer outside -2147483640 to 2147483647, fails the writer.
description yes The INFO header description: one line, with no double quote or backslash.
first_element no Take the first element of a repeated column; an empty list is missing. Default false.
fields no Struct fields tried in order, after first_element. For an entries field, the struct fields of each entry.
required no Fail when no listed column is in any bucket, or a listed struct field is in none of those columns' types. Default true.
shape no scalar, list or entries; see Lists and entries. Default scalar.
source_identity no The source_id, schema_version, release and genome_build of the source; see Source identity.

Each column is extracted before the next is tried, so an empty list or an all-null struct falls back to the next column, as a null does. With first_element or fields, a string value is parsed as JSON first, the form SQLite returns; a string that is not JSON is used as it is. The value left must be a scalar. Annotation fields are filled only for an ALT whose result joined.

String values are percent-encoded so a value cannot break the VCF line or split the per-ALT list: %, ,, ;, =, tab, CR and LF become %25, %2C, %3B, %3D, %09, %0D and %0A, as VCF 4.3 encodes them. A space becomes %20, because many inputs declare VCFv4.2, which forbids whitespace in INFO values. : is left as it is.

A declared ID the input VCF already defines, such as AF, fails the writer, because writing it would replace the input's own values. An ALTAR1_ ID is the exception, since an earlier augmentation wrote it: a header line with the same Number and Type is kept and its values replaced, and a different Number or Type fails.

Without a declaration, the writer emits these default fields. They are not required, so a job without a region or AlphaMissense source leaves them missing:

INFO ID Type Columns Extraction Source
ALTAR1_REGION String region_type regions
ALTAR1_NEAREST_GENE String nearest_genes first element, then gene_name, symbol, name or gene_id regions
ALTAR1_CCRE String ccre_id, then ccre regions
ALTAR1_AM_SCORE Float am_pathogenicity alphamissense
ALTAR1_AM_CLASS String am_class alphamissense

A declaration file must set include_defaults: true keeps the defaults and adds its fields after them, false writes only its fields. This file maps a gnomAD column af_global and the phred field of a struct column cadd to INFO IDs of your choice:

{
  "include_defaults": true,
  "fields": [
    {
      "id": "COHORT_GNOMAD_AF",
      "columns": ["af_global"],
      "type": "Float",
      "description": "gnomAD v4.1.1 genomes global allele frequency per ALT"
    },
    {
      "id": "COHORT_CADD_PHRED",
      "columns": ["cadd"],
      "type": "Float",
      "description": "CADD v1.7 PHRED score per ALT",
      "fields": ["phred"]
    }
  ]
}

Pass it with --annotation-fields-json annotation-fields.json. In Python a declaration is an altar.variants.AnnotationInfoField (provisional). The output manifest records the declarations under annotation_fields, with every key filled in.

Lists and entries

A scalar field has one value per ALT (Number=A). Two shapes carry a repeated column:

  • list: one String per ALT (Number=A) joining every element with &. With fields, each struct element becomes its first non-null listed field. ClinVar's conditions are a list field.
  • entries: one String entry per element of a list of structs (Number=.). An entry is the one-based ALT index, then the listed struct fields, separated by |. The INFO description ends with the format, such as Format: ALT|transcript_id|uniprot_id|protein_variant|am_pathogenicity|am_class. A record with two ALTs and three transcripts has three entries, each naming its ALT.

Both shapes are String. A number is written as Python writes it, and a boolean as 1 or 0. A list or entry value also encodes & as %26 and | as %7C; split on & or | first, then decode each part, for example with urllib.parse.unquote.

Missing values are written as .: a null, empty string or empty list in a Number=A field, a null element or struct field in its place, and an ALT that was not scored, such as a spanning deletion. An ALT with no element has no entry, and a field no ALT of a record has is left out. A scalar String field without a source identity, such as the defaults, writes an empty string for a missing value instead. htslib reads both as missing.

Source identity

A field with a source_identity writes it into its INFO header line:

##INFO=<ID=ALTAR1_GNOMAD_AF,Number=A,Type=Float,Description="gnomAD allele frequency per ALT",Source="org.kundajelab.altar.annotation.gnomad",SchemaVersion="1.0.0",Version="4.1.1-genomes",GenomeBuild="hg38">

Source is the annotation contract, SchemaVersion its version, Version the data release, and GenomeBuild the assembly of the source's variant IDs. The writer fails when GenomeBuild differs from the ingest manifest's reference_build, or when the input already carries the field from a different source.

Source presets

A source binding can ship its own declarations, its preset, from a vcf_info_fields() classmethod on its source class. The run-export command uses each source's preset and attaches the identity the run recorded:

Source type Binding INFO fields
gnomad altar-popfreq ALTAR1_GNOMAD_*, one per contract column, such as ALTAR1_GNOMAD_AF and ALTAR1_GNOMAD_AF_NFE
clinvar altar-clinvar ALTAR1_CLINVAR_*, one per contract column. Conditions, conflicting classifications, molecular consequences and origins are list fields; ALTAR1_CLINVAR_GENES has one entry per gene
phylop100way altar-conservation ALTAR1_PHYLOP100WAY
opentargets_e2g altar-opentargets-e2g ALTAR1_OT_E2G_* for the top gene, its score and the counts; ALTAR1_OT_E2G_LINKS has one entry per link
alphamissense altar-alphamissense ALTAR1_AM_SCORE and ALTAR1_AM_CLASS; ALTAR1_AM_TRANSCRIPTS has one entry per transcript
spliceai altar-spliceai ALTAR1_SPLICEAI_* for the top gene's scores; ALTAR1_SPLICEAI_GENES has one entry per gene

An Open Targets E2G source built with column_prefix names its columns with the prefix, so its preset does not match; pass altar_opentargets_e2g.vcf_info_fields(column_prefix=...) as extra fields instead.

To give your own source a preset, return a tuple of AnnotationInfoField values from vcf_info_fields(). altar.variants.check_annotation_info_fields(fields, contract) checks that each field reads a column of the contract and that its shape and type fit the column. The hook and both names are provisional.

Next