Ingest variants and write augmented VCFs¶
altar.variants ingests a VCF or a variant TSV against a reference genome. It writes every ALT allele with its record
number, the canonical variants to score, and a manifest. After scoring, it writes the results back into the same VCF as
INFO fields, the augmented VCF. Run these steps yourself when your own application keeps a cohort's records, samples and
genotypes beside its scores. altar run reads a VCF itself; for that path see
Give a run its variants.
The ingest and export steps are commands, run with python -m. The functions altar.variants exports are stable,
except the names Provisional names lists. Import from altar.variants
itself, not from the modules below it.
Run the ingest pipeline¶
The pipeline reads a coordinate-sorted VCF, a BGZF-compressed VCF, or a headerless TSV with the columns
chr, pos, ref, alt and an optional fifth ID column. It reads the input through htslib (pysam) a batch of ALT alleles
at a time, so a large callset never sits in memory.
python -m altar.variants.preprocessing.pipeline \
-i input.vcf.gz \
-g genome.fa \
--genome-build hg38 \
-o valid.tsv \
--invalid-out invalid.tsv \
--region-out-dir regions/ \
--occurrence-out-dir ingest/occurrences/ \
--canonical-out-dir ingest/canonical/valid/ \
--invalid-parquet-out-dir ingest/canonical/invalid/ \
--manifest-out ingest/manifest.json \
--header-out ingest/header.vcf
The last five options are optional. Without them, the pipeline writes the same files under an ingest/ directory
beside valid.tsv. -f sets the input format (auto, tsv or vcf); auto, the default, reads the file extension.
--genome-build names the assembly of the input and of the FASTA, such as hg38 or mm10. The default is hg38. A
GRC name is recorded in Altar's spelling, so GRCm38 becomes mm10, and an unknown assembly is rejected before the
input is read. The build becomes the genome of every ingested row and the manifest's reference_build. Score the
cohort with run inputs on the same build. The region-category TSVs come from hg38 annotation data, so on any other
assembly they are empty.
For a multi-sample VCF where only some samples are cases, pass --target-samples-file with one VCF sample ID per line.
An ALT is kept for scoring when at least one selected sample carries it. The manifest records a digest and a count of
the selected samples, not their IDs.
Ingest a callset split across files¶
For a callset split into several VCF files, ingest each file with selected_vcf_ingest. It checks the file and the
sample list against digests you recorded, and skips region filtering:
python -m altar.variants.preprocessing.selected_vcf_ingest \
--input shard-01.vcf.gz --genome genome.fa --genome-build hg38 \
--selected-samples-file case-samples.txt --output-dir shard-01-ingest \
--expected-input-sha256 "$(sha256sum shard-01.vcf.gz | cut -d' ' -f1)" \
--expected-selected-samples-sha256 "$SAMPLES_SHA256"
The sample digest is the hex SHA-256 of the sorted sample IDs as a JSON list.
altar.variants.canonical_sample_selection_digest(samples) computes it (provisional). Once every file's ingest has
written its manifest, merge the canonical variants and drop repeats:
python -m altar.variants.preprocessing.callset_union \
--ingest-dir shard-01-ingest --ingest-dir shard-02-ingest \
--output-dir callset-union
The union rejects an ingest whose outputs changed, or that ran on another build or under another normalization policy. It writes its own manifest last. Neither command decides which samples are cases; you choose them, and the sample digest records the choice.
What the pipeline writes¶
The pipeline writes:
| Output | Contents |
|---|---|
valid.tsv |
Headerless chr, pos, ref, alt, source_variant_id rows, one per scoreable allele. |
invalid.tsv |
One row per rejected allele, with a stable error code. |
regions/ |
One TSV per region category, such as coding or promoter. |
ingest/occurrences/ |
Parquet files with one row per ALT of every record, including unsupported alleles, keyed by (record_ordinal, alt_index). |
ingest/canonical/valid/ |
Parquet files of the scoreable alleles, by canonical variant ID. |
ingest/canonical/invalid/ |
Parquet files of the rejected alleles. |
ingest/header.vcf |
The input's parsed header. |
ingest/manifest.json |
Input and header digests, counts, each Parquet file's row count and digest, the reference build, and the canonicalizer, validation and normalization versions. |
Each Parquet file is renamed into place once complete, beside a success marker with its row count, size and SHA-256
digest. The manifest is written last and marks the whole ingest complete. The Parquet schemas are
OCCURRENCE_ARROW_SCHEMA, CANONICAL_ARROW_SCHEMA and INVALID_ARROW_SCHEMA in altar.variants (provisional).
How records become rows:
- A multi-allelic record gives one occurrence per ALT, in the record's order.
- A repeated record gives repeated occurrences. The canonical ID is
chr:pos:ref:alt, and a consumer selects it distinctly. - The VCF
ID, or the fifth TSV column, becomessource_variant_id; see Custom variant IDs. - A spanning deletion
*, a symbolic structural variant, a breakend, or an ALT equal to REF stays in the occurrences with statusUNSUPPORTEDand an error code. It is not sent to scoring. - The chromosome name is normalized during reference validation. POS stays one-based.
- An allele that passes the reference check is trimmed and left-aligned before its ID is assigned. The canonical
shards and
valid.tsvcarry the normalized locus, and the occurrence keeps the input's spelling in itsoriginal_*columns. The manifest records the policy asnormalization_policy,altar-normalize-v1. - INFO, FORMAT and sample columns are not copied. The input VCF keeps them for the augmented VCF.
Variant identity and normalization explains the reference checks and the normalization rule.
Inputs the pipeline refuses¶
The pipeline refuses these inputs before it writes any output:
- a gVCF: a header declaring
##GVCFBlock*or##ALT=<ID=NON_REF,...>, or, without a declaration, the first<NON_REF>,<*>orEND > POSreference block without a concrete ALT. A symbolic structural variant withENDand a spanning deletion*are not taken for gVCF records. - unsorted VCF records, because the augmented VCF carries a CSI index;
- BCF, with a message to convert it to VCF first.
Load a small input as a DataFrame¶
The DataFrame loaders, read_variants_frame, load_variants and load_variants_vcf, return a whole pandas DataFrame
for a small input. Their variant_id is the canonical ID, and they do not keep the VCF ID. They do not refuse a gVCF:
for VCF input they drop and count symbolic ALTs such as <NON_REF> and any ALT with no
variant key, and keep concrete alleles. A TSV has no such records, so load_variants
and read_variants_frame reject a TSV at the first row with no key and name the row. Every loader rejects a position
that is not ASCII digits. They only trim alleles; validate_variants(frame, genome) checks each row against the
reference and left-aligns the valid ones. Use the pipeline for a large file.
Build and check variant IDs¶
A variant ID is chr:pos:REF:ALT, such as chr1:100:A:G. canonical_variant_id(chromosome, position, ref, alt) builds
one from locus fields, and VariantKey parses and checks one:
from altar.variants import VariantKey, canonical_variant_id
canonical_variant_id("1", 100, "a", "g") # 'chr1:100:A:G'
key = VariantKey.require_canonical("chr1:100:A:G")
key.chromosome, key.position, key.reference_allele, key.alternate_allele # ('chr1', 100, 'A', 'G')
Both are also exported from altar.models and altar.sources, and come from the
altar-identity package that model runtimes import, so every runtime builds the same IDs.
Scoring and assembly reject an ID that is not canonical.
An indel can be written in more than one place. minimal_representation(position, ref, alt) trims the bases REF and ALT
share. left_align(chromosome, position, ref, alt, genome) trims and then shifts an indel left through a repeat, with
genome an open pyfaidx.Fasta of the cohort's build. An annotation lookup trims an ID but never left-aligns it, so an
indel written at the right end of a repeat finds no gnomAD or ClinVar row. Left-align IDs that did not come from the
pipeline or from altar run with a reference before you look them up:
import pyfaidx
from altar.variants import canonical_variant_id, left_align
genome = pyfaidx.Fasta("hg38.fa")
position, ref, alt = left_align("chr1", 107, "AA", "A", genome)
variant_id = canonical_variant_id("chr1", position, ref, alt)
validate_variant(chromosome, position, ref, alt, genome) checks one variant against the reference, and
validate_variants(frame, genome) checks a DataFrame and returns the valid and invalid rows.
Custom variant IDs¶
The VCF ID and the optional fifth TSV column are labels from the input, such as rs123 or a cohort's own IDs. They
are not join keys. The pipeline keeps them as source_variant_id in valid.tsv and the occurrences, where the
augmented VCF can use them. Scores, details and annotations join on the canonical ID, which model runtimes always derive
from chr, pos, ref and alt. The DataFrame loaders do the same and drop the input's IDs; use the pipeline's
outputs when you need them.
The genome build is not part of the ID. It travels with source identities and run provenance, so the same coordinates on two assemblies are never taken for one variant.
Write an augmented VCF¶
The augmented VCF is the input VCF with Altar's results added as INFO fields, aligned with each record's ALT order. It is written from two inputs: the unchanged input VCF, and occurrence buckets that join each ALT to its result row.
BigQueryScoreStore.export_occurrence_bucketsjoins the ingest's occurrence table to the results in SQL and writes the buckets.altar.results.write_occurrence_bucketswrites the same buckets from any store's assembled results (materialize) and the ingest's occurrence files.- After
altar run,python -m altar.variants.export.run_exportandaltar.variants.augment_run_exportbuild the buckets from the run's export and write the VCF in one step; see Write an augmented VCF.
Then write the VCF:
python -m altar.variants.export.augmented_vcf \
--input-vcf input.vcf.gz \
--bucket-dir buckets/ \
--models-json models.json \
--ingest-manifest ingest/manifest.json \
--output-vcf cohort.altar.vcf.gz \
--output-manifest cohort.altar.manifest.json \
--results-parquet-uri gs://bucket/job/results
It streams the input VCF through htslib and writes a BGZF VCF, its CSI index, and a manifest with checksums.
--results-parquet-uri is recorded in the manifest as the results the VCF came from. --annotation-fields-json picks
the annotation INFO fields; see Choose the INFO fields.
The writer names no model. ALTAR1_MAX_ABS_LOGFC reads each model's logfc field, a score several models share.
--include-model-scores adds one ALTAR1_MODEL_SCORE entry per model, holding the fields models.json lists:
{
"score_fields": ["logfc", "jsd", "active_allele_quantile", "in_peak"],
"models": [{"model_id": "...", "name": "...", "column_prefix": "..."}]
}
Use the model manifest's result-schema field names. Each entry is ALT|model|<score_fields...>|prioritized, and the
INFO header records that order. --include-model-scores fails without score_fields. column_prefix is the model's
prefix in the wide export, altar.results.export_column_prefix(model_id, model_name).
Occurrence buckets¶
The bucket directory follows the altar-augmentation-buckets-v1 schema, which the output manifest records as
bucket_schema_version. It holds bucket=<n>/part-*.parquet files with one row per input ALT. The writer reads these
columns, plus the annotation columns its INFO fields name, and ignores the rest:
| Column | Type | Fills | From |
|---|---|---|---|
record_ordinal |
int | alignment | the occurrences (zero-based record number) |
alt_index |
int | alignment | the occurrences (one-based ALT number) |
status, error_code |
str | ALTAR1_STATUS, ALTAR1_ERROR |
the occurrences |
result_variant_id |
str | whether a result joined | the joined result's ID, null when none joined |
prioritized |
bool | ALTAR1_PRIORITIZED |
the assembled result |
<column_prefix>_logfc |
float | ALTAR1_MAX_ABS_LOGFC |
each model's logfc field |
<column_prefix>_<score_field>, <column_prefix>_prioritized |
any | ALTAR1_MODEL_SCORE |
with --include-model-scores |
most_active_celltype |
str | ALTAR1_TOP_MODEL |
an optional display column naming the headline model |
The writer reads the result columns only when result_variant_id is set. PENDING status becomes SCORED when a
result joined and NO_RESULT otherwise. A bucket keeps every annotation column of every joined source with its native
type, so a column from your own source reaches the writer unchanged.
Choose the INFO fields¶
The core fields ALTAR1_STATUS, ALTAR1_ERROR, ALTAR1_PRIORITIZED, ALTAR1_MAX_ABS_LOGFC, ALTAR1_TOP_MODEL and
ALTAR1_MODEL_SCORE are fixed. Annotation fields are declared: each declaration maps one or more bucket columns to one
INFO field.
Declare annotation fields¶
A declaration has these keys:
| Key | Required | Meaning |
|---|---|---|
id |
yes | The INFO ID ([A-Za-z_][A-Za-z0-9_.]*). Not a core field ID, END, or another declaration's ID. |
columns |
yes | Bucket columns tried in order; the first value that is not null or empty after extraction wins. |
type |
yes | Integer, Float or String. A boolean writes 1 or 0. A value that does not convert, or an Integer outside -2147483640 to 2147483647, fails the writer. |
description |
yes | The INFO header description: one line, with no double quote or backslash. |
first_element |
no | Take the first element of a repeated column; an empty list is missing. Default false. |
fields |
no | Struct fields tried in order, after first_element. For an entries field, the struct fields of each entry. |
required |
no | Fail when no listed column is in any bucket, or a listed struct field is in none of those columns' types. Default true. |
shape |
no | scalar, list or entries; see Lists and entries. Default scalar. |
source_identity |
no | The source_id, schema_version, release and genome_build of the source; see Source identity. |
Each column is extracted before the next is tried, so an empty list or an all-null struct falls back to the next column,
as a null does. With first_element or fields, a string value is parsed as JSON first, the form SQLite returns; a
string that is not JSON is used as it is. The value left must be a scalar. Annotation fields are filled only for an ALT
whose result joined.
String values are percent-encoded so a value cannot break the VCF line or split the per-ALT list: %, ,, ;, =,
tab, CR and LF become %25, %2C, %3B, %3D, %09, %0D and %0A, as VCF 4.3 encodes them. A space becomes
%20, because many inputs declare VCFv4.2, which forbids whitespace in INFO values. : is left as it is.
A declared ID the input VCF already defines, such as AF, fails the writer, because writing it would replace the
input's own values. An ALTAR1_ ID is the exception, since an earlier augmentation wrote it: a header line with the
same Number and Type is kept and its values replaced, and a different Number or Type fails.
Without a declaration, the writer emits these default fields. They are not required, so a job without a region or
AlphaMissense source leaves them missing:
| INFO ID | Type | Columns | Extraction | Source |
|---|---|---|---|---|
ALTAR1_REGION |
String | region_type |
regions |
|
ALTAR1_NEAREST_GENE |
String | nearest_genes |
first element, then gene_name, symbol, name or gene_id |
regions |
ALTAR1_CCRE |
String | ccre_id, then ccre |
regions |
|
ALTAR1_AM_SCORE |
Float | am_pathogenicity |
alphamissense |
|
ALTAR1_AM_CLASS |
String | am_class |
alphamissense |
A declaration file must set include_defaults: true keeps the defaults and adds its fields after them, false
writes only its fields. This file maps a gnomAD column af_global and the phred field of a struct column cadd to
INFO IDs of your choice:
{
"include_defaults": true,
"fields": [
{
"id": "COHORT_GNOMAD_AF",
"columns": ["af_global"],
"type": "Float",
"description": "gnomAD v4.1.1 genomes global allele frequency per ALT"
},
{
"id": "COHORT_CADD_PHRED",
"columns": ["cadd"],
"type": "Float",
"description": "CADD v1.7 PHRED score per ALT",
"fields": ["phred"]
}
]
}
Pass it with --annotation-fields-json annotation-fields.json. In Python a declaration is an
altar.variants.AnnotationInfoField (provisional). The output manifest records the declarations under
annotation_fields, with every key filled in.
Lists and entries¶
A scalar field has one value per ALT (Number=A). Two shapes carry a repeated column:
list: oneStringper ALT (Number=A) joining every element with&. Withfields, each struct element becomes its first non-null listed field. ClinVar's conditions are alistfield.entries: oneStringentry per element of a list of structs (Number=.). An entry is the one-based ALT index, then the listed struct fields, separated by|. The INFO description ends with the format, such asFormat: ALT|transcript_id|uniprot_id|protein_variant|am_pathogenicity|am_class. A record with two ALTs and three transcripts has three entries, each naming its ALT.
Both shapes are String. A number is written as Python writes it, and a boolean as 1 or 0. A list or entry value
also encodes & as %26 and | as %7C; split on & or | first, then decode each part, for example with
urllib.parse.unquote.
Missing values are written as .: a null, empty string or empty list in a Number=A field, a null element or struct
field in its place, and an ALT that was not scored, such as a spanning deletion. An ALT with no element has no entry,
and a field no ALT of a record has is left out. A scalar String field without a source identity, such as the
defaults, writes an empty string for a missing value instead. htslib reads both as missing.
Source identity¶
A field with a source_identity writes it into its INFO header line:
##INFO=<ID=ALTAR1_GNOMAD_AF,Number=A,Type=Float,Description="gnomAD allele frequency per ALT",Source="org.kundajelab.altar.annotation.gnomad",SchemaVersion="1.0.0",Version="4.1.1-genomes",GenomeBuild="hg38">
Source is the annotation contract, SchemaVersion its version, Version the data release, and GenomeBuild the
assembly of the source's variant IDs. The writer fails when GenomeBuild differs from the ingest manifest's
reference_build, or when the input already carries the field from a different source.
Source presets¶
A source binding can ship its own declarations, its preset, from a vcf_info_fields() classmethod on its source class.
The run-export command uses each source's preset and attaches the identity the run recorded:
| Source type | Binding | INFO fields |
|---|---|---|
gnomad |
altar-popfreq |
ALTAR1_GNOMAD_*, one per contract column, such as ALTAR1_GNOMAD_AF and ALTAR1_GNOMAD_AF_NFE |
clinvar |
altar-clinvar |
ALTAR1_CLINVAR_*, one per contract column. Conditions, conflicting classifications, molecular consequences and origins are list fields; ALTAR1_CLINVAR_GENES has one entry per gene |
phylop100way |
altar-conservation |
ALTAR1_PHYLOP100WAY |
opentargets_e2g |
altar-opentargets-e2g |
ALTAR1_OT_E2G_* for the top gene, its score and the counts; ALTAR1_OT_E2G_LINKS has one entry per link |
alphamissense |
altar-alphamissense |
ALTAR1_AM_SCORE and ALTAR1_AM_CLASS; ALTAR1_AM_TRANSCRIPTS has one entry per transcript |
spliceai |
altar-spliceai |
ALTAR1_SPLICEAI_* for the top gene's scores; ALTAR1_SPLICEAI_GENES has one entry per gene |
An Open Targets E2G source built with column_prefix names its columns with the prefix, so its preset does not match;
pass altar_opentargets_e2g.vcf_info_fields(column_prefix=...) as extra fields instead.
To give your own source a preset, return a tuple of AnnotationInfoField values from vcf_info_fields().
altar.variants.check_annotation_info_fields(fields, contract) checks that each field reads a column of the contract
and that its shape and type fit the column. The hook and both names are provisional.
Next¶
- Score a cohort scores the ingested variants.
- Give a run its variants names the cohort in a run spec instead.