Skip to content

Variants

altar.variants ingests and validates variants in their canonical form. It also exports the region-annotation contract, REGION_ANNOTATION_CONTRACT, and its implementation, RegionAnnotationSource; see Genomic regions. Names other than the identity primitives load lazily, so importing the contract loads no dataframe or interval-tree library. Ingest variants and write augmented VCFs shows how to use it.

Stable, except the streaming-ingest schemas and the augmented-VCF names, which are provisional. Provisional names lists them.

variants

Canonical variant ingestion, validation, annotation, and export primitives.

The altar.variants namespace is Altar's shared scientific boundary. Model bindings consume these representations rather than defining architecture-specific variant formats.

The region-annotation contract, REGION_ANNOTATION_CONTRACT, and its portable implementation, RegionAnnotationSource, are exported here as well.

Allele normalization is the pair minimal_representation (reference-free, from altar_identity) and left_align (against an open reference). Ingestion applies them before a variant's canonical key is assigned, so a caller keying variants outside the ingest pipeline applies the same pair to get the same keys.

The streaming ingest's Arrow schemas (OCCURRENCE_ARROW_SCHEMA, CANONICAL_ARROW_SCHEMA, INVALID_ARROW_SCHEMA) and canonical_sample_selection_digest, the hash of a VCF sample selection, are provisional and listed in __provisional__.

The augmented-VCF names are provisional too. AnnotationInfoField declares one INFO field, which a source binding uses for its preset, and check_annotation_info_fields checks a preset against the binding's contract. augment_run_export writes an augmented VCF from an altar run export.

Names other than the identity primitives resolve lazily (PEP 562): importing the package or the contract loads neither the dataframe stack (pandas, pyfaidx) nor the region lookup stack (intervaltree, ncls). The defining module loads when one of its names is first accessed.

REGION_ANNOTATION_CONTRACT module-attribute

REGION_ANNOTATION_CONTRACT = AnnotationContract(
    source_id=REGION_ANNOTATION_SOURCE_ID,
    schema_version="1.0.0",
    columns=(
        AnnotationColumn(
            name="region_type",
            dtype="str",
            label="Region Type",
        ),
        AnnotationColumn(
            name="region_labels",
            dtype="str",
            label="Region Labels",
            repeated=True,
        ),
        AnnotationColumn(
            name="region_gene_ids",
            dtype="str",
            label="Region Gene IDs",
            repeated=True,
        ),
        AnnotationColumn(
            name="is_coding", dtype="bool", label="Coding"
        ),
        AnnotationColumn(
            name="in_promoter",
            dtype="bool",
            label="In Promoter",
        ),
        AnnotationColumn(
            name="nearest_genes",
            dtype="struct",
            label="Nearest Genes",
            repeated=True,
            fields=(
                AnnotationColumn(
                    name="gene_name",
                    dtype="str",
                    label="Gene Name",
                ),
                AnnotationColumn(
                    name="gene_id",
                    dtype="str",
                    label="Gene ID",
                ),
                AnnotationColumn(
                    name="distance",
                    dtype="int",
                    label="Distance",
                ),
                AnnotationColumn(
                    name="type",
                    dtype="str",
                    label="Gene Type",
                ),
            ),
        ),
        AnnotationColumn(
            name="gene_within_100kb",
            dtype="bool",
            label="Gene Within 100 kb",
        ),
        AnnotationColumn(
            name="ccre_id", dtype="str", label="cCRE ID"
        ),
        AnnotationColumn(
            name="ccre_group",
            dtype="str",
            label="cCRE Group",
        ),
    ),
)

The region-annotation columns, version 1.0.0. ChromBPNet's prioritization depends on region_type.

VariantIdentityError

Bases: ValueError

A variant key is malformed, non-canonical, or disagrees with its locus.

VariantKey dataclass

VariantKey(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
)

Canonical one-based biallelic key serialized as chr:pos:ref:alt.

Fields follow the rules in this module's documentation. The class normalizes spelling but does not normalize biological representation. Callers must left-align and trim indels against the selected reference before constructing a key when equivalence across representations matters.

variant_id property

variant_id: str

Return the portable score, detail, and annotation join key.

parse classmethod

parse(variant_id: str) -> VariantKey

Parse and normalize a four-field textual key.

require_canonical classmethod

require_canonical(variant_id: str) -> VariantKey

Parse variant_id and reject aliases instead of silently rewriting them.

from_fields classmethod

from_fields(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
) -> VariantKey

Construct a canonical key from decomposed one-based locus fields.

RegionAnnotationSource

RegionAnnotationSource(
    data_dir: str | PathLike[str] | None = None,
    genome_build: str = REGION_GENOME_BUILD,
)

Bases: AnnotationSource

Where each variant falls on hg38, from region data you build: gene region, promoter, nearest genes, cCRE.

It computes the org.kundajelab.altar.annotation.regions columns. Build the region data once, as Build another release describes, into a directory you name with data_dir or $ALTAR_VARIANTS_DATA_DIR.

data_dir is the annotation data directory, resolved once at construction: the argument, else $ALTAR_VARIANTS_DATA_DIR, else the installed package's data directory. It must hold region_annotations.parquet, ccres.dnatree, and region_provenance.json; a directory without its own gene_df.tsv uses the shipped one. The data is GRCh38, so genome_build must be "hg38" (or its alias "GRCh38"). Materialization therefore joins this source only to hg38 variants; variants on any other assembly get no region columns.

identity derives the release from the inputs and logic version region_provenance.json records, after checking each file's SHA-256 against it, and annotate runs that check before it loads anything. The cCRE tree is unpickled only from bytes with the recorded digest. These checks catch drift and accidents; they do not protect against someone who can write the data directory. A variant on a chromosome without protein-coding genes in the gene table, such as an unplaced contig, has no row.

The source is registered under altar.annotation_sources as regions, and a run spec declares it with the constructor's arguments as options (data_dir, and genome_build).

from_options classmethod

from_options(
    *,
    data_dir: str | PathLike[str] | None = None,
    genome_build: str = REGION_GENOME_BUILD,
) -> Self

Build the source from a run spec's options: data_dir, the region data directory, and genome_build.

Without data_dir, the source reads $ALTAR_VARIANTS_DATA_DIR, else the installed package's data directory. genome_build must be "hg38", the default. Another option raises TypeError.

annotation_contract classmethod

annotation_contract() -> AnnotationContract

Return REGION_ANNOTATION_CONTRACT, which every instance implements.

identity

Return the contract identity, with the release named by the verified provenance manifest.

annotate async

annotate(
    variant_ids: Sequence[str],
) -> dict[str, dict[str, Any]]

Return the region columns of each variant, keyed by the ID as passed.

Each variant is looked up at the span of its minimal representation (minimal_variant_key), as the allelic sources look it up, so chr1:100:CA:CT is placed at chr1:101.

ValidationErrorReason

Bases: Enum

Reasons why a variant validation might fail.

Generic validation never emits ALT_LENGTH_MISMATCH or ALT_MISMATCH. They described a model input window that no longer belongs to core validation and remain defined so persisted reason values keep parsing.

ValidationResult dataclass

ValidationResult(
    is_valid: bool,
    chro: str | None = None,
    error_reason: ValidationErrorReason | None = None,
    error_message: str | None = None,
)

Result of validating a single variant.

AnnotationInfoField dataclass

AnnotationInfoField(
    id: str,
    columns: tuple[str, ...],
    value_type: str,
    description: str,
    first_element: bool = False,
    fields: tuple[str, ...] = (),
    required: bool = True,
    shape: str = "scalar",
    source_identity: AnnotationSourceIdentity | None = None,
)

A caller-declared INFO field filled from bucket annotation columns.

id is the INFO ID; it must not name a core field or END. columns are bucket columns tried in order: the first column whose extracted value is not null or empty wins, so a renamed column can fall back to its legacy name. value_type is the VCF Type: Integer, Float, or String. A boolean renders as 1 or 0.

shape is scalar (the default), list or entries; the module docstring describes each. A list or entries field is a String.

For a scalar field, first_element takes the first element of a repeated column, and fields then reads the first non-missing of these struct fields, so nearest_genes[0].gene_name is first_element=True, fields=("gene_name",). For a list field, fields reads the first non-missing struct field of each element. For an entries field, fields names the struct fields of each entry, in order. A string value is parsed as JSON first when first_element or fields is set or the shape is not scalar: a JSON-backed store returns that form. A string that is not JSON is used as is.

A required field fails the writer when none of its columns is in any bucket part, or when one of its struct fields is in no struct type those columns have, which catches a misspelled column or field. A field that is not required is left missing instead.

source_identity names the contract, data release and genome build of the source the columns come from. The writer puts it in the field's INFO header line and checks its build against the input's.

number property

number: str

The VCF Number: . for an entries field, A otherwise.

info_field

info_field() -> InfoField

Return the INFO header line this field writes.

to_dict

to_dict() -> dict[str, Any]

Return the declaration in its JSON form, with every key filled in.

from_dict classmethod

from_dict(data: Mapping[str, Any]) -> AnnotationInfoField

Read one declaration in the to_dict form.

Only id, columns, type and description are required.

canonical_chromosome

canonical_chromosome(chromosome: str) -> str

Return the canonical chr-prefixed spelling used in portable keys.

Primary chromosomes accept common aliases (1/chr1/CHR01 and M/MT/chrM) in any case. Other reference contigs keep their exact spelling after an optional case-insensitive chr prefix is normalized, so CHRUn_KI270302v1 becomes chrUn_KI270302v1 but chrun_KI270302v1 stays as written. Non-ASCII text, whitespace inside the name, and characters outside the VCF contig-name set (including :) raise VariantIdentityError. This is a naming rule only; it does not claim that contigs from different assemblies are interchangeable.

canonical_variant_id

canonical_variant_id(
    chromosome: str,
    position: int,
    reference_allele: str,
    alternate_allele: str,
) -> str

Return the canonical text key for decomposed one-based locus fields.

minimal_representation

minimal_representation(
    position: int,
    reference_allele: str,
    alternate_allele: str,
) -> tuple[int, str, str]

Trim the bases that REF and ALT share, keeping at least one base in each.

The shared suffix is removed first, then the shared prefix, and position advances by one for each prefix base removed, so the result names the same change at the same locus: (10, "CAT", "CT") becomes (10, "CA", "C") and (10, "TCA", "TGA") becomes (11, "C", "G"). Trimming the suffix first leaves an indel anchored on the base before it, the VCF convention. A SNV, and alleles that share no end base (A/AT, AT/A), come back unchanged. Alleles are spelled as the key spells them: trimmed of ASCII whitespace and uppercased.

The alleles must have a key: each is [ACGTN]+ and they differ, like VariantKey. Symbolic (<DEL>), spanning-deletion *, breakend and missing alleles raise VariantIdentityError rather than being trimmed as if their characters were bases, and so do equal alleles, which would otherwise collapse to one shared base that is not a variant. The position is not validated beyond being an int, so a one-based or zero-based position stays in its own convention.

Trimming is a pure function of the alleles. Left alignment, which shifts an indel through a repeat using the reference sequence, is a separate step that ingestion applies around this one.

left_align

left_align(
    chrom: str, pos: int, ref: str, alt: str, genome: Fasta
) -> tuple[int, str, str]

Normalize one variant against the reference: minimal representation, then left alignment.

pos is one-based. The alleles are first reduced to their minimal representation (altar_identity.minimal_representation). An indel is then shifted left through any repeat: while the last bases of REF and ALT agree and a reference base precedes the variant, that base is prepended to both alleles and their last base dropped, moving the variant one position left. A final minimal representation drops the anchor bases the shift carried along. Equal-length alleles (SNVs and MNVs) are never shifted. This is what lets an indel written at the right end of a homopolymer find its gnomAD or ClinVar record: the gnomAD and ClinVar builders only trim, and rely on gnomAD and ClinVar publishing left-aligned VCFs.

The chromosome is read under its canonical name, the one validate_variant uses, and reference bases are uppercased, so soft-masking does not stop a shift. The shift stops at a reference base outside ACGT (such as N), so a variant validate_variant accepted keeps alleles it accepts. REF must equal the reference at pos, which validate_variant checks; the shift assumes it and does not re-check. A chromosome absent from the FASTA raises KeyError.

Parameters:

Name Type Description Default
chrom str

Chromosome, under any alias canonical_chromosome accepts

required
pos int

Position (1-based)

required
ref str

Reference allele

required
alt str

Alternate allele

required
genome Fasta

Open genome file handle

required

Returns:

Type Description
tuple[int, str, str]

The normalized (pos, ref, alt), with alleles spelled as the variant key spells them

load_variants

load_variants(variants_loc: str) -> DataFrame

Load a headerless TSV and derive canonical score keys from its locus fields.

An optional fifth input column is a source identifier. The occurrence-aware streaming pipeline preserves it as source_variant_id; this compatibility DataFrame returns the canonical five-column scoring schema instead.

load_variants_vcf

load_variants_vcf(path: str) -> DataFrame

Parse VCF ALT occurrences and derive canonical locus-based score keys.

read_variants_frame

read_variants_frame(
    path: str, fmt: str = "auto"
) -> DataFrame

Load variants into the canonical table, dispatching on file format.

This is the format-neutral entry point; load_variants (TSV) and load_variants_vcf (VCF/gVCF) are the per-format adapters it composes. It loads the whole file into memory; altar_identity.read_variants, by contrast, streams VariantKey values from Altar's container variant file.

Parameters:

Name Type Description Default
path str

Input variants file.

required
fmt str

One of "auto", "tsv", "vcf". "auto" routes .vcf/.vcf.gz (and .gvcf) to the VCF parser and everything else to load_variants; .bcf raises a clear error.

'auto'

Returns:

Type Description
DataFrame

DataFrame with columns chr, pos, ref, alt, variant_id.

validate_variant

validate_variant(
    chro: str, pos: int, ref: str, alt: str, genome: Fasta
) -> ValidationResult

Check one variant against the reference genome, independent of any model.

The chromosome must be a primary chromosome (1-22, X, Y or M, under any alias that canonical_chromosome accepts) and present in the FASTA under its canonical name. REF and ALT must be non-empty uppercase ACGT and differ, REF must lie inside the contig, and REF must equal the reference bases at pos. Soft-masked (lowercase) reference bases match. Whether a model's input window fits around the variant is that model's concern, not part of this check.

To check many variants against one genome, call ReferenceChecker(genome).validate instead. It returns the same results and looks each chromosome up only once.

Parameters:

Name Type Description Default
chro str

Chromosome

required
pos int

Position (0-based)

required
ref str

Reference allele

required
alt str

Alternate allele

required
genome Fasta

Open genome file handle

required

Returns:

Type Description
ValidationResult

ValidationResult with the canonical chromosome, and error details if invalid

validate_variants

validate_variants(
    variants_df: DataFrame, genome: Fasta
) -> tuple[DataFrame, DataFrame]

Validate variants with validate_variant and return valid and invalid dataframes.

A valid row's chr, pos, ref and alt are normalized: the canonical chromosome and the left_align representation of the alleles. Invalid rows keep their input fields.

Parameters:

Name Type Description Default
variants_df DataFrame

DataFrame with variant information

required
genome Fasta

Open genome file handle

required

Returns:

Type Description
DataFrame

Tuple of (valid_variants_df, invalid_variants_df)

DataFrame

Invalid variants have 'error_reason' and 'error_message' columns added.

check_annotation_info_fields

check_annotation_info_fields(
    fields: Sequence[AnnotationInfoField],
    contract: AnnotationContract,
    *,
    column_prefix: str = "",
) -> None

Raise AugmentationError when fields do not match the columns of contract.

Each field must read only columns the contract declares, named with column_prefix when the source adds one. Its shape and type must fit each column:

  • a scalar column renders as a scalar field of the column's type;
  • a repeated scalar column renders as a list field;
  • a repeated record column renders as an entries field, or as a list or scalar field that selects record fields.

Every record field a field names must exist, and IDs must be distinct.

augment_run_export

augment_run_export(
    export_dir: str | Path,
    input_vcf: str | Path,
    output_vcf: str | Path,
    *,
    reference_fasta: str | Path | None = None,
    ingest_dir: str | Path | None = None,
    work_dir: str | Path | None = None,
    output_manifest: str | Path | None = None,
    annotation_fields: Sequence[AnnotationInfoField] = (),
    genome_build: str | None = None,
    bucket_records: int = DEFAULT_BUCKET_RECORDS,
) -> dict[str, Any]

Write output_vcf, the VCF the run read with the export's values as INFO fields, and return a summary.

  • export_dir is a Parquet altar run export over input_vcf.
  • reference_fasta is the FASTA the run named as [variants] reference_fasta. The VCF is ingested against it into work_dir, a temporary directory by default. Pass ingest_dir instead to reuse an ingest of this VCF that holds manifest.json and occurrences/.
  • annotation_fields are added after the sources' presets. A default field whose ID a preset declares, such as ALTAR1_AM_SCORE, is dropped.
  • genome_build is needed only when the export records none.

The output manifest goes to output_manifest, by default beside the VCF as <name>.manifest.json. The returned summary holds that manifest under manifest, the sources without a preset under sources_without_preset, and the join counts under occurrences.

canonical_sample_selection_digest

canonical_sample_selection_digest(
    samples: Sequence[str],
) -> str

Hash a set of VCF sample IDs independently of their supplied order.