Variants¶
altar.variants ingests and validates variants in their canonical form. It also exports the region-annotation
contract, REGION_ANNOTATION_CONTRACT, and its implementation, RegionAnnotationSource; see
Genomic regions. Names other than the identity primitives load lazily, so importing
the contract loads no dataframe or interval-tree library. Ingest variants and write augmented VCFs shows
how to use it.
Stable, except the streaming-ingest schemas and the augmented-VCF names, which are provisional. Provisional names lists them.
variants
¶
Canonical variant ingestion, validation, annotation, and export primitives.
The altar.variants namespace is Altar's shared scientific boundary. Model
bindings consume these representations rather than defining architecture-specific
variant formats.
The region-annotation contract, REGION_ANNOTATION_CONTRACT, and its portable
implementation, RegionAnnotationSource, are exported here as well.
Allele normalization is the pair minimal_representation (reference-free, from
altar_identity) and left_align (against an open reference). Ingestion applies
them before a variant's canonical key is assigned, so a caller keying variants
outside the ingest pipeline applies the same pair to get the same keys.
The streaming ingest's Arrow schemas (OCCURRENCE_ARROW_SCHEMA, CANONICAL_ARROW_SCHEMA,
INVALID_ARROW_SCHEMA) and canonical_sample_selection_digest, the hash of a VCF sample selection, are
provisional and listed in __provisional__.
The augmented-VCF names are provisional too. AnnotationInfoField declares one INFO field, which a source binding
uses for its preset, and check_annotation_info_fields checks a preset against the binding's contract.
augment_run_export writes an augmented VCF from an altar run export.
Names other than the identity primitives resolve lazily (PEP 562): importing the package or the contract loads neither the dataframe stack (pandas, pyfaidx) nor the region lookup stack (intervaltree, ncls). The defining module loads when one of its names is first accessed.
REGION_ANNOTATION_CONTRACT
module-attribute
¶
REGION_ANNOTATION_CONTRACT = AnnotationContract(
source_id=REGION_ANNOTATION_SOURCE_ID,
schema_version="1.0.0",
columns=(
AnnotationColumn(
name="region_type",
dtype="str",
label="Region Type",
),
AnnotationColumn(
name="region_labels",
dtype="str",
label="Region Labels",
repeated=True,
),
AnnotationColumn(
name="region_gene_ids",
dtype="str",
label="Region Gene IDs",
repeated=True,
),
AnnotationColumn(
name="is_coding", dtype="bool", label="Coding"
),
AnnotationColumn(
name="in_promoter",
dtype="bool",
label="In Promoter",
),
AnnotationColumn(
name="nearest_genes",
dtype="struct",
label="Nearest Genes",
repeated=True,
fields=(
AnnotationColumn(
name="gene_name",
dtype="str",
label="Gene Name",
),
AnnotationColumn(
name="gene_id",
dtype="str",
label="Gene ID",
),
AnnotationColumn(
name="distance",
dtype="int",
label="Distance",
),
AnnotationColumn(
name="type",
dtype="str",
label="Gene Type",
),
),
),
AnnotationColumn(
name="gene_within_100kb",
dtype="bool",
label="Gene Within 100 kb",
),
AnnotationColumn(
name="ccre_id", dtype="str", label="cCRE ID"
),
AnnotationColumn(
name="ccre_group",
dtype="str",
label="cCRE Group",
),
),
)
The region-annotation columns, version 1.0.0. ChromBPNet's prioritization depends on region_type.
VariantIdentityError
¶
Bases: ValueError
A variant key is malformed, non-canonical, or disagrees with its locus.
VariantKey
dataclass
¶
Canonical one-based biallelic key serialized as chr:pos:ref:alt.
Fields follow the rules in this module's documentation. The class normalizes spelling but does not normalize biological representation. Callers must left-align and trim indels against the selected reference before constructing a key when equivalence across representations matters.
parse
classmethod
¶
parse(variant_id: str) -> VariantKey
Parse and normalize a four-field textual key.
require_canonical
classmethod
¶
require_canonical(variant_id: str) -> VariantKey
Parse variant_id and reject aliases instead of silently rewriting them.
from_fields
classmethod
¶
from_fields(
chromosome: str,
position: int,
reference_allele: str,
alternate_allele: str,
) -> VariantKey
Construct a canonical key from decomposed one-based locus fields.
RegionAnnotationSource
¶
RegionAnnotationSource(
data_dir: str | PathLike[str] | None = None,
genome_build: str = REGION_GENOME_BUILD,
)
Bases: AnnotationSource
Where each variant falls on hg38, from region data you build: gene region, promoter, nearest genes, cCRE.
It computes the org.kundajelab.altar.annotation.regions columns. Build the region data once, as
Build another release
describes, into a directory you name with data_dir or $ALTAR_VARIANTS_DATA_DIR.
data_dir is the annotation data directory, resolved once at construction: the argument, else
$ALTAR_VARIANTS_DATA_DIR, else the installed package's data directory. It must hold
region_annotations.parquet, ccres.dnatree, and region_provenance.json; a directory without its
own gene_df.tsv uses the shipped one. The data is GRCh38, so genome_build must be "hg38" (or its alias
"GRCh38"). Materialization therefore joins this source only to hg38 variants; variants on any other
assembly get no region columns.
identity derives the release from the inputs and logic version region_provenance.json records,
after checking each file's SHA-256 against it, and annotate runs that check before it loads anything.
The cCRE tree is unpickled only from bytes with the recorded digest. These checks catch drift and accidents;
they do not protect against someone who can write the data directory. A variant on a chromosome without
protein-coding genes in the gene table, such as an unplaced contig, has no row.
The source is registered under altar.annotation_sources as regions, and a run spec declares it with the
constructor's arguments as options (data_dir, and genome_build).
from_options
classmethod
¶
from_options(
*,
data_dir: str | PathLike[str] | None = None,
genome_build: str = REGION_GENOME_BUILD,
) -> Self
Build the source from a run spec's options: data_dir, the region data directory, and genome_build.
Without data_dir, the source reads $ALTAR_VARIANTS_DATA_DIR, else the installed package's data
directory. genome_build must be "hg38", the default. Another option raises TypeError.
annotation_contract
classmethod
¶
annotation_contract() -> AnnotationContract
Return REGION_ANNOTATION_CONTRACT, which every instance implements.
identity
¶
identity() -> AnnotationSourceIdentity
Return the contract identity, with the release named by the verified provenance manifest.
annotate
async
¶
Return the region columns of each variant, keyed by the ID as passed.
Each variant is looked up at the span of its minimal representation (minimal_variant_key), as the
allelic sources look it up, so chr1:100:CA:CT is placed at chr1:101.
ValidationErrorReason
¶
Bases: Enum
Reasons why a variant validation might fail.
Generic validation never emits ALT_LENGTH_MISMATCH or ALT_MISMATCH. They described a model input
window that no longer belongs to core validation and remain defined so persisted reason values keep parsing.
ValidationResult
dataclass
¶
ValidationResult(
is_valid: bool,
chro: str | None = None,
error_reason: ValidationErrorReason | None = None,
error_message: str | None = None,
)
Result of validating a single variant.
AnnotationInfoField
dataclass
¶
AnnotationInfoField(
id: str,
columns: tuple[str, ...],
value_type: str,
description: str,
first_element: bool = False,
fields: tuple[str, ...] = (),
required: bool = True,
shape: str = "scalar",
source_identity: AnnotationSourceIdentity | None = None,
)
A caller-declared INFO field filled from bucket annotation columns.
id is the INFO ID; it must not name a core field or END. columns are bucket columns tried in order:
the first column whose extracted value is not null or empty wins, so a renamed column can fall back to its
legacy name. value_type is the VCF Type: Integer, Float, or String. A boolean renders as 1 or 0.
shape is scalar (the default), list or entries; the module docstring describes each. A list or
entries field is a String.
For a scalar field, first_element takes the first element of a repeated column, and fields then reads
the first non-missing of these struct fields, so nearest_genes[0].gene_name is
first_element=True, fields=("gene_name",). For a list field, fields reads the first non-missing struct
field of each element. For an entries field, fields names the struct fields of each entry, in order.
A string value is parsed as JSON first when first_element or fields is set or the shape is not scalar:
a JSON-backed store returns that form. A string that is not JSON is used as is.
A required field fails the writer when none of its columns is in any bucket part, or when one of its
struct fields is in no struct type those columns have, which catches a misspelled column or field. A
field that is not required is left missing instead.
source_identity names the contract, data release and genome build of the source the columns come from.
The writer puts it in the field's INFO header line and checks its build against the input's.
to_dict
¶
Return the declaration in its JSON form, with every key filled in.
from_dict
classmethod
¶
from_dict(data: Mapping[str, Any]) -> AnnotationInfoField
Read one declaration in the to_dict form.
Only id, columns, type and description are required.
canonical_chromosome
¶
Return the canonical chr-prefixed spelling used in portable keys.
Primary chromosomes accept common aliases (1/chr1/CHR01 and
M/MT/chrM) in any case. Other reference contigs keep their exact
spelling after an optional case-insensitive chr prefix is normalized, so
CHRUn_KI270302v1 becomes chrUn_KI270302v1 but chrun_KI270302v1
stays as written. Non-ASCII text, whitespace inside the name, and characters
outside the VCF contig-name set (including :) raise
VariantIdentityError. This is a naming rule only; it does not claim that
contigs from different assemblies are interchangeable.
canonical_variant_id
¶
canonical_variant_id(
chromosome: str,
position: int,
reference_allele: str,
alternate_allele: str,
) -> str
Return the canonical text key for decomposed one-based locus fields.
minimal_representation
¶
minimal_representation(
position: int,
reference_allele: str,
alternate_allele: str,
) -> tuple[int, str, str]
Trim the bases that REF and ALT share, keeping at least one base in each.
The shared suffix is removed first, then the shared prefix, and position
advances by one for each prefix base removed, so the result names the same
change at the same locus: (10, "CAT", "CT") becomes (10, "CA", "C")
and (10, "TCA", "TGA") becomes (11, "C", "G"). Trimming the suffix
first leaves an indel anchored on the base before it, the VCF convention. A
SNV, and alleles that share no end base (A/AT, AT/A), come
back unchanged. Alleles are spelled as the key spells them: trimmed of ASCII
whitespace and uppercased.
The alleles must have a key: each is [ACGTN]+ and they differ, like
VariantKey. Symbolic (<DEL>), spanning-deletion *, breakend and
missing alleles raise VariantIdentityError rather than being trimmed as if
their characters were bases, and so do equal alleles, which would otherwise
collapse to one shared base that is not a variant. The position is not
validated beyond being an int, so a one-based or zero-based position
stays in its own convention.
Trimming is a pure function of the alleles. Left alignment, which shifts an indel through a repeat using the reference sequence, is a separate step that ingestion applies around this one.
left_align
¶
Normalize one variant against the reference: minimal representation, then left alignment.
pos is one-based. The alleles are first reduced to their minimal representation
(altar_identity.minimal_representation). An indel is then shifted left through any repeat: while the
last bases of REF and ALT agree and a reference base precedes the variant, that base is prepended to both
alleles and their last base dropped, moving the variant one position left. A final minimal representation
drops the anchor bases the shift carried along. Equal-length alleles (SNVs and MNVs) are never shifted.
This is what lets an indel written at the right end of a homopolymer find its gnomAD or ClinVar record:
the gnomAD and ClinVar builders only trim, and rely on gnomAD and ClinVar publishing left-aligned VCFs.
The chromosome is read under its canonical name, the one validate_variant uses, and reference bases
are uppercased, so soft-masking does not stop a shift. The shift stops at a reference base outside ACGT
(such as N), so a variant validate_variant accepted keeps alleles it accepts. REF must equal the
reference at pos, which validate_variant checks; the shift assumes it and does not re-check. A
chromosome absent from the FASTA raises KeyError.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
chrom
|
str
|
Chromosome, under any alias |
required |
pos
|
int
|
Position (1-based) |
required |
ref
|
str
|
Reference allele |
required |
alt
|
str
|
Alternate allele |
required |
genome
|
Fasta
|
Open genome file handle |
required |
Returns:
| Type | Description |
|---|---|
tuple[int, str, str]
|
The normalized |
load_variants
¶
Load a headerless TSV and derive canonical score keys from its locus fields.
An optional fifth input column is a source identifier. The occurrence-aware
streaming pipeline preserves it as source_variant_id; this compatibility
DataFrame returns the canonical five-column scoring schema instead.
load_variants_vcf
¶
Parse VCF ALT occurrences and derive canonical locus-based score keys.
read_variants_frame
¶
Load variants into the canonical table, dispatching on file format.
This is the format-neutral entry point; load_variants (TSV) and
load_variants_vcf (VCF/gVCF) are the per-format adapters it composes.
It loads the whole file into memory; altar_identity.read_variants, by
contrast, streams VariantKey values from Altar's container variant file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
Input variants file. |
required |
fmt
|
str
|
One of |
'auto'
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame with columns |
validate_variant
¶
validate_variant(
chro: str, pos: int, ref: str, alt: str, genome: Fasta
) -> ValidationResult
Check one variant against the reference genome, independent of any model.
The chromosome must be a primary chromosome (1-22, X, Y or M, under any alias that
canonical_chromosome accepts) and present in the FASTA under its canonical name. REF and
ALT must be non-empty uppercase ACGT and differ, REF must lie inside the contig, and REF must equal the
reference bases at pos. Soft-masked (lowercase) reference bases match. Whether a model's
input window fits around the variant is that model's concern, not part of this check.
To check many variants against one genome, call ReferenceChecker(genome).validate instead. It returns
the same results and looks each chromosome up only once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
chro
|
str
|
Chromosome |
required |
pos
|
int
|
Position (0-based) |
required |
ref
|
str
|
Reference allele |
required |
alt
|
str
|
Alternate allele |
required |
genome
|
Fasta
|
Open genome file handle |
required |
Returns:
| Type | Description |
|---|---|
ValidationResult
|
ValidationResult with the canonical chromosome, and error details if invalid |
validate_variants
¶
Validate variants with validate_variant and return valid and invalid dataframes.
A valid row's chr, pos, ref and alt are normalized: the canonical chromosome and the
left_align representation of the alleles. Invalid rows keep their input fields.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
variants_df
|
DataFrame
|
DataFrame with variant information |
required |
genome
|
Fasta
|
Open genome file handle |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
Tuple of (valid_variants_df, invalid_variants_df) |
DataFrame
|
Invalid variants have 'error_reason' and 'error_message' columns added. |
check_annotation_info_fields
¶
check_annotation_info_fields(
fields: Sequence[AnnotationInfoField],
contract: AnnotationContract,
*,
column_prefix: str = "",
) -> None
Raise AugmentationError when fields do not match the columns of contract.
Each field must read only columns the contract declares, named with column_prefix when the source adds one.
Its shape and type must fit each column:
- a scalar column renders as a
scalarfield of the column's type; - a repeated scalar column renders as a
listfield; - a repeated record column renders as an
entriesfield, or as alistorscalarfield that selects record fields.
Every record field a field names must exist, and IDs must be distinct.
augment_run_export
¶
augment_run_export(
export_dir: str | Path,
input_vcf: str | Path,
output_vcf: str | Path,
*,
reference_fasta: str | Path | None = None,
ingest_dir: str | Path | None = None,
work_dir: str | Path | None = None,
output_manifest: str | Path | None = None,
annotation_fields: Sequence[AnnotationInfoField] = (),
genome_build: str | None = None,
bucket_records: int = DEFAULT_BUCKET_RECORDS,
) -> dict[str, Any]
Write output_vcf, the VCF the run read with the export's values as INFO fields, and return a summary.
export_diris a Parquetaltar runexport overinput_vcf.reference_fastais the FASTA the run named as[variants] reference_fasta. The VCF is ingested against it intowork_dir, a temporary directory by default. Passingest_dirinstead to reuse an ingest of this VCF that holdsmanifest.jsonandoccurrences/.annotation_fieldsare added after the sources' presets. A default field whose ID a preset declares, such asALTAR1_AM_SCORE, is dropped.genome_buildis needed only when the export records none.
The output manifest goes to output_manifest, by default beside the VCF as <name>.manifest.json. The
returned summary holds that manifest under manifest, the sources without a preset under
sources_without_preset, and the join counts under occurrences.
canonical_sample_selection_digest
¶
Hash a set of VCF sample IDs independently of their supplied order.