Skip to content

Run driver

altar.run is the Python form of the altar run, altar prepare and altar interpret commands. run() reads a run spec, scores the cohort's cache misses with every model, annotates it with every source, and writes the export. parse_spec() and load_spec() turn a mapping or a TOML or JSON file into a RunSpec.

Stable, except validate(), prepare(), interpret() and the summaries they return, which are provisional. Provisional names lists them.

import asyncio

from altar.run import run

summary = asyncio.run(run("annotate.toml"))
print(summary.to_dict()["export_uri"])
from altar.run import interpret

summary = asyncio.run(interpret("interpret.toml"))
print(summary.products_uri, summary.failed)

The run spec reference lists every section and key, and Use Altar from Python shows the driver in a program. altar spec-schema prints the JSON Schema of a spec.

run

One-call reference driver: a declarative run spec resolved through the plugin registries.

A run is a mapping (TOML or JSON) that names a cohort, the models that score it and the annotation sources that annotate it, and the adapters that execute and store their results. Every adapter axis resolves independently through its entry-point registry by name, so a model, an execution backend, a score store, and an annotation source are chosen without knowing each other (invariant 1). :func:run ingests the cohort, prepares and executes every cache-miss batch of every model through :func:altar.scoring.score_cohort, and exports the materialized results. The export reads each model's scores with the lineage resolution its preparation admitted the cohort with (ScoreReusePolicy.exactly), so it reads the same lineages the cache checks did, including equivalent lineages its binding declares, even when every variant was a hit and the run registered nothing. It is a one-shot driver, not a scheduler: it retries nothing beyond what the engine does, and holds no queue.

A spec without models annotates: it ingests the cohort and exports every candidate variant with its sources' columns, and needs no [backend]. It needs at least one source, declared or passed in code; a spec with neither models nor sources is rejected. Its [store] is optional: without one the export reads through an in-memory SQLite store, so the run writes no store file. A spec with models and no [store] keeps their scores in SQLite at <cache_dir>/scores.sqlite, so later runs reuse them. Without models, a variant is prioritized only when a source that prioritizes, such as AlphaMissense, flags it.

A run keeps its files in two directories ([run]). The run directory, work_dir, defaults to ./<job_id> and holds what only this run reads: the candidate file, the files staged for and collected from this job's containers (staging/jobs/<job_id>/ and results/jobs/<job_id>/), and the export. Deleting it loses no reusable work. The cache, cache_dir, defaults to $ALTAR_CACHE_DIR or ./.altar and holds what a later run reads: the files altar prepare publishes under staging/, the default score store, the default artifact and interpretation stores, interpretation checkpoints, and the local backend's default data_root, backend-data. It is created only when something is written there. A container file whose logical path starts with jobs/ goes to the run directory, and every other one to the cache. A default run directory that already exists, holds files, and is not an earlier run's is refused, so a job_id never writes into an unrelated directory.

The run normalizes every variant it ingests, from a VCF or a canonical TSV, into <work_dir>/variants/<job_id>.candidates.tsv. With [variants] reference_fasta each variant is left-aligned against it (:func:altar.variants.left_align), so a deletion written at the right end of a repeat matches the form gnomAD and ClinVar store; a variant on a contig the FASTA lacks, or whose REF the reference does not hold, is only trimmed, and a model's preparation sets it aside. Without a reference each variant is trimmed to its minimal representation (:func:altar_identity.minimal_representation). Two rows that normalize to one variant are one candidate. A TSV's fields are separated by tabs or spaces, and its optional label is not carried over. The rows a lookup may miss are counted, by reason, as :class:PotentialMiss entries of :attr:RunSummary.potential_misses (provisional): indels only trimmed for want of a reference, which gnomAD and ClinVar key left-aligned; with a reference, variants whose REF it does not hold or whose contig it lacks; and, in a run with sources, variants on contigs other than chr1-22, X, Y and M, which the annotation datasets do not cover. They are advisory: the run still exports every variant.

[run] genome_build is the assembly of the cohort's coordinates, normalized to Altar's spelling (GRCh38 is read as hg38; see altar.models.canonical_genome_build). Every model scores the cohort on that assembly, whatever assembly the model was trained on. Before any source is built, each declared source's class is checked against it: a class whose supported_genome_builds leave the run's build out cannot serve it with any options, so the run stops and names every such source, advising to remove it or lift the cohort over. Each built source's identity().genome_build is then checked too, so an hg19 cohort is never annotated with hg38 data; only a source whose class supports the run's build is advised to point at its data on that build. The UCSC and GRC spellings of one assembly compare equal. A source that declares no identity is not checked here; the export treats it as serving hg38, so it annotates no cohort on any other assembly.

[run] genome_build is checked against each model's run inputs the same way, before any source is built: a model whose supported_genome_builds leave the run's build out is named in the same error. Then each model's configuration, run inputs and output selection are parsed, and a binding's validation error becomes one line per problem that names the spec setting to change. A model whose plan runs in containers needs a [backend], and the files its plan reads from the platform, which a preparation step writes into <cache_dir>/staging, must be there. A [backend] whose data_root names a directory the run cannot create or write is refused. With the local backend each model's image is looked up with docker image inspect, then docker manifest inspect; an image in neither place is a warning. All of this runs before the cohort is read.

A spec declares one [backend], or several [backends.<name>] in place of it. A container model runs on the spec's only backend, or on the one its backend names, which it must name when there are several; its preparation and interpretation run there too. Its pool replaces the GPU pool its binding's tasks ask for, for every stage or per stage (score, preprocess, interpret), and leaves a task that asks for no GPU on the CPU. A pool must be one the backend defines (ExecutionBackend.gpu_pools), and a binding's own pool too unless the backend is cpu_only; both are checked before anything is submitted. Neither choice is part of the model's identity, so stored scores are reused wherever they were computed; :attr:ModelRunSummary.compute and the provenance record where this run computed. Models on different backends score one after another.

A VCF is read one line at a time. An ALT that names no sequence a variant key can hold (., *, a symbolic <DEL>, a breakend, ALT equal to REF, or an IUPAC code) is skipped and counted by reason in :attr:RunSummary.dropped_alleles. A malformed row, a VCF that would be read as a TSV, and a position that is not ASCII digits are :class:RunSpecError, naming the row. Hashing a large reference FASTA is logged with its digest. A supplied [variants] reference_digest is verified against the file before normalization or cache lookup. The first check hashes the file and writes a verification record beside it; later runs accept the unchanged file from that record without reading it.

[models.<name>] resources names a registry entry, "<registry>/<id>" such as "legnet/wtc11", optionally pinned as "<registry>/<id>@sha256:<hex>" (see altar.registries). The entry gives the model's type, which the section may repeat, and its saved configuration, which configuration may extend or override field by field. [variants] reference names a genome of the genome registry by its id, such as "grch38/ncbi/no-alt-analysis-set", or an alias, such as "hg38", or an entry of another registry as "<registry>/<id>", instead of a path; a name whose first segment is no registry's is a genome's id. [registries] adds registries to the installed ones and those of ALTAR_REGISTRIES: a name and its index's URI, or {uri, connection, allow_private_hosts}. A registry hosted elsewhere is read over https://, or from gs://, s3:// or hf:// anonymously unless it names a connection, and may name only such addresses on public hosts; allow_private_hosts, on the table or on its [connections.<name>], admits an intranet one (altar.registries). Every name resolves in the cache, [run] cache_dir, to the files altar resources fetch put there; a run never downloads, and an entry not fetched stops the run, and validate, with the command that fetches it. Before the binding builds its configuration, each file's uri is replaced with its path in the cache. Identity reads digests only, so the scores are those of the same configuration read from anywhere. A named genome's digest is its entry's, the verification record the fetch wrote accepts the FASTA without hashing it, and it must be on [run] genome_build. An entry's prepared files are put where altar prepare would have published them, with its record, so the model scores without altar prepare; its binding checks them as it checks those (ModelPlugin.prepared_run_inputs). The registry, its URI, the entry's id and its digest are recorded in ModelRunSummary.registry and the provenance, never in identity.

resources may also be a list of entries, which the section expands into one model per entry, in order. Each model's key is the entry's EntryName.model_key, <registry>-<id> in lowercase with . as _, such as lab-encsr296uhq: the key a one-entry section [models.lab-encsr296uhq] resources = "lab/ENCSR296UHQ" has, so the two forms reuse each other's scores, and moving an entry between sections changes nothing stored. The section's other keys apply to every model; a type must be every entry's, model_name names one model and is refused, and an entry named twice, or whose key is another section's, is refused. The section's key names it in logs, ModelSectionSummary, ModelRunSummary.section and the provenance, never in identity. An entry whose training_reference is not the run's reference is warned about, once per section with the counts.

Each model whose plugin reads the reference genome gets the FASTA, identified by its digest, as its run input genome, and that digest is part of its scores' identity. A model that reads none (ModelPlugin.reads_reference_genome), such as AlphaGenome, gets no reference, so adding or changing reference_fasta leaves its identity alone. The FASTA is verified when a model reads it or the spec asserts reference_digest, including annotation-only runs. Static validation does not read it.

The run logs one line per stage to the altar.run logger, starting with the spec section it belongs to, such as [variants]: the cohort read, each source opened (with its release and where it reads), each model scored, and the export written, with counts and elapsed times. A model's line counts the variants it set aside by reason and the rows its store added. When the run ends it logs the skipped VCF alleles and the potential misses as warnings. :func:validate (provisional) runs every check that needs no variant, building the sources and the backend but reading no cohort and writing no file, and returns the :class:RunPlan it would run.

The :class:RunSummary records each source's identity, the release and build it served, as a :class:SourceRunSummary (provisional, listed in __provisional__). A local export directory holds the summary as _run_summary.json beside the exported files; the leading underscore keeps a Parquet reader that opens the whole directory from taking it for a data file.

:attr:RunSummary.provenance records, for each model, the identity the run requested and the lineages its exported rows came from, and the release and build of each source the export joins. The export carries the same JSON: a Parquet file in its key-value metadata under altar.provenance, and a TSV in _provenance.json beside it.

A wide export, the default, also has _columns.tsv beside its table, the column manifest: one row per column, naming the model, plugin, result schema and field each model column holds (altar.results.wide_column_manifest). [export] layout = "long" (provisional) writes a models table, a variants table and one score table per result schema in place of the wide table, with _provenance.json beside them in either format (altar.results.render_long_export). Every layout reads the store's result buckets (ScoreStore.result_buckets), so its rows are in genomic order: chromosomes 1, 2, …, X, Y, M, then other contigs, each by position.

Sections, with run and variants required, and at least one of models and sources::

[run]            job_id, genome_build, work_dir (default ./<job_id>), cache_dir (default $ALTAR_CACHE_DIR or
                 ./.altar)
[variants]       path (a VCF or a canonical headerless TSV), format, reference | reference_fasta,
                 reference_digest (with reference_fasta only)
[registries]     <name> = "<uri>" or {uri, connection}: registries beside the installed ones
[models.<name>]  type, resources (an entry, or a list) | resource_manifest, configuration, run_inputs, output,
                 max_variants_per_batch, model_name, backend, pool     (none: annotate only)
[backend]        type, options, poll_interval_s, timeout, cpu_only     (needed by container plans only)
[backends.<name>] the keys of [backend], in place of it: named backends each model chooses from
[connections.<name>] type, options
[storage]        type, options, connection                              (default: local)
[store]          type, options, connection   (default: sqlite at <cache_dir>/scores.sqlite with models, in
                                  memory without; a new sqlite store keeps lineages unless options
                                  say lineages = false)
[detail_store]   type, options, connection   (default: the store's type on the store's connection, built
                                  only for a model that declares detail tables; a store with a table_id
                                  keeps them in <table_id>__details, a sqlite store at scores.sqlite keeps
                                  them in scores.details.sqlite beside it, and an in-memory one in memory)
[artifact_store] type, options   (required when a model's output selects an artifact; built only then)
[sources.<name>] type, options, connection
[export]         format (parquet | tsv; default parquet; csv is a deprecated name for tsv), layout (wide |
                 long; default wide), destination, prioritized_only, bucket_rows (variants the store hands
                 the writers at a time; default: about 128 MiB of results)

Relative paths, both the spec's own and those inside adapter options and model configuration, which Altar passes through verbatim, are resolved against the working directory the run is started from.

An adapter section's options are the keyword arguments of the registered class's constructor, so a section can declare only an adapter whose constructor takes values a mapping can carry. The shipped execution backends and score and detail stores do: local (data_root, slots, container_user) and sqlite (path, and lineages for the score store). A store section that names a connection is built through the store's from_options instead, which builds its client from the connection: the bigquery score store takes table_id, lineage_table_id and primary_stage_rows, and the bigquery detail store takes table_id, each reaching BigQuery through connection.bigquery_client() and closing that client when the run ends. An annotation source is built through its AnnotationSource.from_options hook instead, since most take a backend and a release record: the shipped regions source takes data_dir, and the AlphaMissense, SpliceAI, GPN-Star, population-frequency, and ClinVar bindings take dataset, the path of a dataset their builder wrote, whose metadata names the release and genome build, plus the source's own settings. The population-frequency and ClinVar sources read their publisher's copy of the latest release when no dataset is given, or a BigQuery table through a connection, which a bigquery store joins into its own SQL. A source without the hook, such as the core sqlite source, which takes AnnotationColumn values, and the bigquery source adapter, which takes a client, is built in code and passed to :func:run as annotation_sources; a sources section that names one fails before any scoring starts. Likewise an inline model whose execution needs an injected client takes it through runtimes, keyed by the model's spec name; a spec is durable and never carries a credential or a client.

A [connections.<name>] section declares one way of reaching object storage or a warehouse, a registered :class:~altar.connections.Connection built from its options: gcp (Application Default Credentials, with project, anonymous, impersonate_service_account and requester_pays) or fsspec (storage_options and protocols). A sources section, [storage], [store] or [detail_store] names one with connection, and the driver passes the live object to it as the connection keyword: to the source's or the store's from_options, or to the storage adapter's constructor. Only an adapter that names a connection parameter takes one; a section that gives connection to any other adapter, or names a connection the spec does not declare, is rejected, and a spec never names a client or a credential. Two connections let one run read a dataset where its publisher hosts it, anonymously (public, for a source whose from_options takes a connection), and keep its own files in the user's bucket, billed to their project and read as their service account (lab)::

[connections.public]
type = "gcp"
options = { anonymous = true }

[connections.lab]
type = "gcp"
[connections.lab.options]
project = "my-lab-project"
impersonate_service_account = "reader@my-lab-project.iam.gserviceaccount.com"

[storage]
type = "fsspec"
connection = "lab"

The same lab connection keeps scores and details in BigQuery, billed to its project. A BigQueryScoreStore can only join sources that read BigQuery in its SQL (SqlAnnotationSource), such as a population-frequency source over an owned table. The run writes either export format itself, from the store's result buckets::

[store]
type = "bigquery"
connection = "lab"
options = { table_id = "my-lab-project.altar.scores" }

[detail_store]
type = "bigquery"
connection = "lab"
options = { table_id = "my-lab-project.altar.details" }

[export]
format = "parquet"

Without the [detail_store] section, the details go to my-lab-project.altar.scores__details through the store's lab connection.

A host supplies a connection built in code, such as one holding a secret credential, through run(..., connections={"lab": ...}). Every connection a run uses is declared in the spec, and the host's object wins: it replaces the spec's declaration of the same name, which is then not built, so the spec still runs on its own with its declared, ambient-credential default. Injecting a name the spec does not declare is an error rather than a silent addition. A connection is operational: it never enters a source's, a model's, or a cache's identity, so reading the same data through another connection reuses every score.

The [models.<name>] key is the model's stable model_id and the column prefix of its export block, so keep it the same across runs of the same model instance for its scores to be cache hits.

A spec with models that keeps its scores in a new SQLite store, in memory or at a path that holds no scores yet, gets lineages = true unless its options set lineages. A model whose setting changes then gets a lineage of its own beside the old one. A store file that already holds scores is opened in the mode it was created with and never converted; a lineages option that disagrees with it is refused. In a store without lineages, a changed setting stops the run with a :class:RunSpecError that names both ways on: a new [models.<name>] key, or a new store with lineages = true.

A model's output is its :class:~altar.models.OutputSelection, and naming an artifact the binding declares in its outputs (Sei's chromatin_profile_effects, for example) makes the run produce and store it. Such a run needs an [artifact_store]; a spec that selects an artifact without one is rejected before anything is ingested. The shipped storage store copies files through the run's [storage] and is built with it; its root_uri defaults to <cache_dir>/artifacts and its SQLite index_path to <cache_dir>/artifacts/index.sqlite, so a later run's artifacts are cache hits too::

[models.sei.output]
outputs = ["scores", "sequence_class_effects", "chromatin_profile_effects"]
reducer = "official_nucleosome_adjusted_sequence_class_effect_v1"

[artifact_store]
type = "storage"

EntryText module-attribute

EntryText = Annotated[str, Field(min_length=1)]

A registry entry as a spec writes it, <registry>/<id> or <registry>/<id>@sha256:<hex>.

BackendName module-attribute

BackendName = Annotated[
    str, Field(pattern="^[A-Za-z_][A-Za-z0-9_-]*$")
]

A [backends.<name>] key: a simple identifier, which may also hold hyphens.

RegistryValue module-attribute

RegistryValue = (
    Annotated[str, Field(min_length=1)] | RegistrySettings
)

A registry's index URI, or {uri, connection, allow_private_hosts}.

RunSpecError

Bases: ValueError

The run spec is malformed, or names something the installed registries cannot build.

RunSettings

Bases: _SpecModel

Identity and directories of one run.

work_dir is the run directory: the candidate file, this job's container files and the export, which no later run reads. Without it the run directory is ./<job_id> (:attr:run_dir). cache_dir holds everything a later run reuses: prepared model files, the default score, artifact and interpretation stores, interpretation checkpoints, and the local backend's default data_root. Without it, it is $ALTAR_CACHE_DIR, or ./.altar when that is not set, read when the spec is parsed.

run_dir property

run_dir: Path

Return the run directory: work_dir, or ./<job_id> when the spec leaves it out.

VariantSettings

Bases: _ReferenceSettings

The cohort: a VCF or canonical TSV, and the reference its models score against.

ModelSettings

Bases: _SpecModel

One model instance: its registry type, durable configuration, and per-run inputs.

type class-attribute instance-attribute

type: Annotated[str, Field(min_length=1)] | None = None

The registered model type, such as LEGNET; a registry entry named with resources gives it.

configuration class-attribute instance-attribute

configuration: dict[str, Any] = {}

The binding's configuration fields; with resources or resource_manifest, the ones to add or change.

resources class-attribute instance-attribute

resources: EntryText | list[EntryText] | None = None

A registry entry, <registry>/<id> such as legnet/wtc11, fetched with altar resources fetch, or a list of them, which the section expands into one model per entry.

It is an alternative to resource_manifest: the entry gives the model's type and saved configuration. It may be pinned to one entry document as <registry>/<id>@sha256:<hex>. A list's models are keyed by each entry's EntryName.model_key, such as lab-encsr296uhq, and share the section's other keys.

resource_manifest class-attribute instance-attribute

resource_manifest: Path | None = None

A binding-written resource manifest; configuration then holds the remaining settings.

registry_entry property

registry_entry: FetchedEntry | None

The registry entry resources named, once the run resolved it in the cache; None before or without.

section property

section: str | None

The [models.<section>] whose resources list this model was expanded from; None for its own.

entries property

entries: tuple[EntryName, ...]

The entries resources names, parsed: one for a string, each of a list, none without.

run_inputs class-attribute instance-attribute

run_inputs: dict[str, Any] = {}

Binding run inputs beyond genome_build and genome, which the run supplies.

backend class-attribute instance-attribute

backend: BackendName | None = None

The [backends.<name>] the model's containers run on: scoring, altar prepare and altar interpret.

It is required only when the spec declares several backends; with one, every model runs on it. Where a model runs is not part of its identity, so its stored scores are reused whichever backend computed them.

pool class-attribute instance-attribute

pool: PoolName | dict[str, PoolName] | None = None

The GPU pool the model's GPU tasks ask for, in place of the binding's default; a name its backend defines.

A table sets it per stage, by the operations the binding declares (score, preprocess, interpret): pool = {score = "a100"}. A stage the table leaves out keeps the binding's pool, and a task that asks for no GPU stays on the CPU. Like backend, it is not part of the model's identity.

RegistrySettings

Bases: _SpecModel

A [registries] value given as a table: the index's URI, the connection that reads it, and its opt-in.

allow_private_hosts lets a registry hosted elsewhere name hosts on a private network, such as an intranet mirror; without it, a fetch refuses any host that resolves to a private, loopback or link-local address.

AdapterSettings

Bases: _SpecModel

A registered adapter and the keyword arguments of its constructor.

ConnectionSettings

Bases: AdapterSettings

A [connections.<name>] section: its connection type, the type's options, and its private-host opt-in.

allow_private_hosts lets every registry hosted elsewhere that reads through this connection name hosts on a private network, as a [registries] table's own key does. It is not passed to the connection.

ConnectedAdapterSettings

Bases: AdapterSettings

An adapter that may read through one of the spec's named connections.

connection class-attribute instance-attribute

connection: Name | None = None

A [connections.<name>] key; the live connection is passed to the adapter as connection.

BackendSettings

Bases: AdapterSettings

The execution backend, how often the run checks its tasks, and how long it waits for them.

timeout class-attribute instance-attribute

timeout: Annotated[
    float,
    Field(gt=0),
    WithJsonSchema(
        {
            "anyOf": [
                {"type": "number", "exclusiveMinimum": 0},
                {
                    "type": "string",
                    "pattern": "^\\s*\\d+(\\.\\d+)?\\s*[smhdSMHD]\\s*$",
                },
            ],
            "default": 86400,
        }
    ),
] = 86400.0

How long the run waits on one set of tasks, in seconds: one model's batch, or one wave of workflow stages for altar prepare and altar interpret. A spec may write it as text such as "90m", "24h" or "2d". Tasks still open after it are cancelled, where the backend can cancel a task, and the run stops.

cpu_only class-attribute instance-attribute

cpu_only: bool = False

Clear every task's GPU request so a plan built for accelerators runs on a CPU-only host.

ExportSettings

Bases: _SpecModel

The export written after every model has scored.

format is parquet (the default), which keeps repeated and struct columns as lists and structs, or tsv, gzipped tab-separated text with those columns as JSON text. csv is a deprecated name for tsv: the file was always tab-separated, and a run that names it logs a warning.

layout is wide (the default), one table with a row per variant and a block of columns per model, with its column manifest _columns.tsv beside it, or long (provisional), a models table, a variants table and one score table per result schema with a row per variant and model (see altar.results.write_long_export).

bucket_rows class-attribute instance-attribute

bucket_rows: Annotated[int, Field(ge=1)] | None = None

Variants per result bucket the export reads from the store at a time; by default a size of about 128 MiB.

InterpretationSettings

Bases: _SpecModel

One interpretation request: a model of the spec, the method to run, the target it explains, and the variants.

model names a [models.<name>] section, whose type, configuration, resource manifest and run inputs the interpretation uses; that key is also the model ID its products are stored under. method names one method and methods several, each run as its own operation; a method runs every method it requires. target defaults to the methods' only target. variants lists canonical variant IDs to interpret instead of the run's cohort. configuration and run_inputs are added to the model's own, for settings only interpretation reads, such as ChromBPNet's motifs.

requested_methods property

requested_methods: tuple[str, ...]

Return the methods the section asks for, in order.

InterpretationStoreSettings

Bases: _SpecModel

Where interpretation products are kept: the reference store, a file tree and its SQLite index.

options takes root_uri, the tree the product files are copied to through the run's [storage] (default <cache_dir>/interpretations), and index_path, the SQLite index (default index.sqlite in that directory). storage is the only type until an interpretation-store entry-point group exists.

RunSpec

Bases: _SpecModel

A complete, validated run.

models class-attribute instance-attribute

models: dict[Name, ModelSettings] = {}

The models to score; a spec without any annotates the cohort through its sources and exports those.

backend class-attribute instance-attribute

backend: BackendSettings | None = None

The run's one backend, which every model's containers run on.

backends class-attribute instance-attribute

backends: dict[BackendName, BackendSettings] = {}

Named backends, in place of backend; each model names the one it runs on with backend.

connections class-attribute instance-attribute

connections: dict[Name, ConnectionSettings] = {}

Named ways of reaching object storage and warehouses, which storage and sources refer to.

registries class-attribute instance-attribute

registries: dict[Name, RegistryValue] = {}

Registries of models and genomes beside the installed ones, by name: an index's URI or a RegistrySettings.

store class-attribute instance-attribute

store: ConnectedAdapterSettings | None = None

Where scores are kept, and the store the export reads through. A spec with models that leaves it out keeps them in SQLite at <cache_dir>/scores.sqlite; one without models gets an in-memory SQLite store, so the run writes no store file.

artifact_store class-attribute instance-attribute

artifact_store: AdapterSettings | None = None

Where the artifacts a model's output selects are stored; required when any model selects one.

interpretations class-attribute instance-attribute

interpretations: dict[Name, InterpretationSettings] = {}

Interpretation requests, which altar interpret runs (provisional); altar run leaves them alone.

interpretation_store class-attribute instance-attribute

interpretation_store: InterpretationStoreSettings | None = (
    None
)

Where altar interpret keeps interpretation products (provisional); defaults to the reference store.

ModelRunSummary dataclass

ModelRunSummary(
    name: str,
    plugin_id: str,
    candidate_count: int,
    duplicate_count: int,
    ineligible_count: int,
    cache_hit_count: int,
    scored_count: int,
    batch_count: int,
    added: int,
    skipped: int,
    ineligible_report: str | None,
    details: tuple[DetailRunResult, ...] = (),
    artifacts: tuple[ArtifactRunResult, ...] = (),
    variants_without_rows: int = 0,
    ineligible_reasons: dict[str, int] = dict(),
    compute: dict[str, Any] | None = None,
    registry: dict[str, str] | None = None,
    section: str | None = None,
)

Preparation counts and persisted rows for one model of the run.

variants_without_rows class-attribute instance-attribute

variants_without_rows: int = 0

Variants sent to the model that got no primary row, summed over its batches.

ineligible_reasons class-attribute instance-attribute

ineligible_reasons: dict[str, int] = field(
    default_factory=dict
)

ineligible_count by reason, from the ineligible-variant report: an IneligibilityReason value, or for unsupported_variant_class the variant's class, as unsupported_variant_class:deletion.

compute class-attribute instance-attribute

compute: dict[str, Any] | None = None

Where the model's containers ran, or None for a model that runs inline (provisional): backend, the [backends.<name>] key (None for the spec's single [backend]), its registered type, pools, the GPU pools its scoring tasks ask for after the model's pool, and cpu_only, whether the backend dropped those GPU requests. It records where this run computed, not where cached scores came from.

registry class-attribute instance-attribute

registry: dict[str, str] | None = None

The registry entry resources named (provisional): registry, its name; uri, where its index is; id, the entry's id; and entry_sha256, the entry document's digest. None for a model named another way. Like compute, it is not part of the model's identity.

section class-attribute instance-attribute

section: str | None = None

The [models.<section>] whose resources list the model was expanded from (provisional); None for a model with a section of its own. Not part of the model's identity: its key, name, is the entry's.

ModelSectionSummary dataclass

ModelSectionSummary(
    name: str, count: int, models: tuple[str, ...]
)

A [models.<name>] section whose resources list expanded into models (provisional).

name instance-attribute

name: str

The section's key.

count instance-attribute

count: int

How many models it expanded into.

models instance-attribute

models: tuple[str, ...]

Their keys, in the list's order.

SourceRunSummary dataclass

SourceRunSummary(
    name: str,
    section: str | None,
    identity: dict[str, str] | None,
)

One annotation source the run exported: the release and genome build it served, as it declares them.

name instance-attribute

name: str

The source's name.

section instance-attribute

section: str | None

The spec section that declared it, sources.<name>, or None for one passed in code.

identity instance-attribute

identity: dict[str, str] | None

AnnotationSourceIdentity.to_dict(): contract, schema version, release and genome build, or None for a source that declares no identity.

PotentialMiss dataclass

PotentialMiss(
    reason: str, count: int, example: str, message: str
)

Variants of the cohort that a lookup may miss or read at the wrong place; advisory, they are still exported.

reason is one of:

  • indel_not_left_aligned: an indel only trimmed, because the spec gives no [variants] reference_fasta to left-align it against; gnomAD and ClinVar key indels left-aligned, so one written further right in a repeat is not found;
  • reference_mismatch: REF is not the reference's bases at that position, so the variant was only trimmed, and a lookup reads it as written (only checked with a reference);
  • contig_not_in_reference: the reference FASTA has no contig of that name (only checked with a reference);
  • contig_not_primary: a contig other than chr1-22, X, Y and M, such as an alt or unplaced contig, which the annotation datasets do not cover, so its source columns are empty (only counted when the run has annotation sources).

count instance-attribute

count: int

How many of the cohort's input rows have this reason.

example instance-attribute

example: str

One of them, as chr:pos:ref:alt with the alleles as the input wrote them.

message instance-attribute

message: str

What happened and what to do about it, in a sentence or two; altar run warns of it in a few lines.

RunSummary dataclass

RunSummary(
    job_id: str,
    genome_build: str,
    candidate_variants: str,
    variant_count: int,
    models: tuple[ModelRunSummary, ...],
    export_uri: str,
    sources: tuple[SourceRunSummary, ...] = (),
    potential_misses: tuple[PotentialMiss, ...] = (),
    dropped_alleles: dict[str, int] = dict(),
    provenance: dict[str, Any] = dict(),
    sections: tuple[ModelSectionSummary, ...] = (),
    export_layout: str = "wide",
    column_manifest_uri: str | None = None,
    export_tables: dict[str, str] = dict(),
)

What :func:run ingested, scored, annotated, and exported.

potential_misses class-attribute instance-attribute

potential_misses: tuple[PotentialMiss, ...] = ()

Variants a lookup may miss, by reason (see :class:PotentialMiss); empty when none was found.

dropped_alleles class-attribute instance-attribute

dropped_alleles: dict[str, int] = field(
    default_factory=dict
)

VCF ALT alleles that name no sequence a variant key can hold, by reason, so they are not in the export: no_alternate (.), spanning_deletion (*), symbolic (<DEL>), breakend, reference_only (ALT equal to REF), and not_acgtn (an IUPAC code or another base outside ACGTN).

provenance class-attribute instance-attribute

provenance: dict[str, Any] = field(default_factory=dict)

What produced the export's model scores and source columns, as JSON-compatible values.

The export carries the same payload. It is {"format": 1, "models": {<spec name>: {...}}, "sources": {<source name>: {...}}}. Each source the export joins maps to its identity, as :class:SourceRunSummary records it (source_id, schema_version, release and genome_build), or to null for a source that declares none. Each model's entry holds:

  • model_name, column_prefix (the prefix of its export columns) and plugin_id;
  • requested: the identity this run asked for, which it stamps on the rows it writes. It holds the lineage_id, plugin_version, configuration_hash, scientific_configuration_hash, reducer, output_selection_hash, result_schema_hash, runtimes (each name, kind and identity, such as a container image digest), genome_build, outputs and result_schema_version;
  • reuse: the lineages the run read. scope is the widest step, current, equivalent or accepted. lineages lists each lineage_id and step in preference order, or is null for a store that keeps no lineages. equivalence_rules_id names the binding's equivalence rules when they ran;
  • this_run: cache_hits, the variants the run found already scored, and rows_added, the rows it wrote;
  • compute: where this run ran the model's containers, as :attr:ModelRunSummary.compute records it (backend, type, pools and cpu_only), or null for an inline model. It is not part of the model's identity: a cache hit's producer may have run anywhere;
  • registry: the registry entry that named the model, as ModelRunSummary.registry records it, and section: the section whose resources list it was expanded from, or null for either. Neither is part of the model's identity;
  • producers: one entry per lineage the exported rows came from, in preference order. Each has its lineage_id, step, rows (the exported rows under it) and recorded_runs, the identities the store recorded as writing under it, empty for an imported lineage. The store does not record which of them wrote a given row. A store that keeps no lineages gives "unknown".

A cache hit is never attributed to this run: its producer is the lineage it was stored under.

export says how the export is laid out: layout, wide or long, and column_manifest, the file beside a wide export that describes its columns, _columns.tsv, or null for a long export.

sections class-attribute instance-attribute

sections: tuple[ModelSectionSummary, ...] = ()

Each section whose resources list expanded into several of models, with its count (provisional).

export_layout class-attribute instance-attribute

export_layout: str = 'wide'

wide or long, as [export] layout set it.

column_manifest_uri class-attribute instance-attribute

column_manifest_uri: str | None = None

Where a wide export's column manifest, _columns.tsv, was written: one row per column of the table, naming the model, plugin, result schema and field of each model column (provisional). None for a long export.

export_tables class-attribute instance-attribute

export_tables: dict[str, str] = field(default_factory=dict)

A long export's tables, each table's path within the export (models, variants, scores/<name>) to the directory it was written to (provisional). Empty for a wide export.

to_dict

to_dict() -> dict[str, Any]

Return the summary as JSON-compatible values.

RunPlan dataclass

RunPlan(
    job_id: str,
    genome_build: str,
    work_dir: str,
    cache_dir: str,
    variants: str,
    reference_fasta: str | None,
    models: dict[str, str],
    sources: tuple[SourceRunSummary, ...],
    store: str,
    export_format: str,
    export_uri: str,
    compute: dict[str, dict[str, Any] | None] = dict(),
    sections: tuple[ModelSectionSummary, ...] = (),
    export_layout: str = "wide",
)

What a valid spec would run, from :func:validate: nothing was ingested, scored, or written.

work_dir instance-attribute

work_dir: str

The run directory: [run] work_dir, or <job_id> when the spec leaves it out.

cache_dir instance-attribute

cache_dir: str

The cache, [run] cache_dir, where reusable files are kept; a run creates it only to write there.

models instance-attribute

models: dict[str, str]

Each model's spec name and the plugin_id of the binding that scores it.

sources instance-attribute

sources: tuple[SourceRunSummary, ...]

Every annotation source, built and checked against genome_build, with the release it serves.

store instance-attribute

store: str

The score store's registered type; for a spec without [store], sqlite (<cache_dir>/scores.sqlite) with models and sqlite (in memory) without.

export_format instance-attribute

export_format: str

parquet or tsv; a spec's deprecated csv is reported as tsv.

compute class-attribute instance-attribute

compute: dict[str, dict[str, Any] | None] = field(
    default_factory=dict
)

Each model's backend and GPU pools, as :attr:ModelRunSummary.compute records them; None for an inline model (provisional).

sections class-attribute instance-attribute

sections: tuple[ModelSectionSummary, ...] = ()

Each section whose resources list expanded into several of models, with its count (provisional).

export_layout class-attribute instance-attribute

export_layout: str = 'wide'

wide or long, as [export] layout sets it.

to_dict

to_dict() -> dict[str, Any]

Return the plan as JSON-compatible values.

ModelPreparationSummary dataclass

ModelPreparationSummary(
    name: str,
    plugin_id: str,
    completed_stages: tuple[str, ...],
    output_uris: tuple[str, ...],
    compute: dict[str, Any] | None = None,
)

One model's completed preparation stages and collected output locations (provisional).

compute class-attribute instance-attribute

compute: dict[str, Any] | None = None

The backend preparation ran on and the GPU pools its tasks asked for, as ModelRunSummary.compute records them.

PreparationSummary dataclass

PreparationSummary(
    job_id: str, models: tuple[ModelPreparationSummary, ...]
)

The model artifacts prepared under a run's staging directory (provisional).

to_dict

to_dict() -> dict[str, Any]

Return JSON-compatible preparation results.

InterpretationProduct dataclass

InterpretationProduct(
    section: str,
    model_id: str,
    method: str,
    variant_id: str,
    product: str,
    shard: str,
    run_id: str,
    operation_id: str,
    interpretation_id: str,
    uri: str,
    row: int,
    chrom: str,
    start: int,
    end: int,
    anchor: int,
    media_type: str,
    digest: str,
)

One stored product file that describes a variant: where its bytes are, which row, and what made it.

method is the requested method of the section's operation. operation_id and run_id name the operation and execution whose commit made the file visible, which may be an earlier request's: its operation file names its input variants. interpretation_id names the method run, and its descriptor in the operation file names the model resources, runtimes and genome. row is the variant's row in the file, and chrom, start, end and anchor give the 0-based, half-open reference span the row covers and the offset of the variant's first base in it. shard holds the file's labels, such as allele=ref.

InterpretationOperationSummary dataclass

InterpretationOperationSummary(
    name: str,
    model_id: str,
    plugin_id: str,
    method: str,
    target: str,
    operation_id: str,
    interpretation_ids: dict[str, str],
    operation_uri: str,
    variant_count: int,
    cached_count: int,
    executed_count: int,
    run_id: str | None,
    resumed: bool,
    committed: tuple[str, ...],
    skipped: dict[str, str],
    failure: str | None,
    missing: tuple[str, ...],
    product_files: int,
)

One operation interpret ran or read: one method of an [interpretations.<name>] section.

operation_id names the request: the method chain's interpretation IDs, by method in interpretation_ids, and the input variants. operation_uri is its JSON file, with every descriptor and the variants. cached_count counts the variants whose products were stored before, and executed_count those the execution run_id ran; run_id is None when nothing ran. committed and skipped say which methods that execution stored, failure is the workflow's error, and missing lists the variants that still have no complete set of products.

complete property

complete: bool

Return whether every requested variant has every product of the method chain.

InterpretationsSummary dataclass

InterpretationsSummary(
    job_id: str,
    genome_build: str,
    store: dict[str, str],
    products_uri: str | None,
    operations: tuple[InterpretationOperationSummary, ...],
    products: tuple[InterpretationProduct, ...] = (),
)

What :func:interpret ran or read, and where it wrote the listing of the stored products.

products_uri is products.tsv, one row per :class:InterpretationProduct, or None for a read limited to some variants, which writes no listing. operations holds one :class:InterpretationOperationSummary per requested method of each section, and products the listed rows. store names the interpretation store's root_uri and index_path.

failed property

failed: tuple[str, ...]

Return <section>.<method> for each operation that left a variant without its products.

to_dict

to_dict(*, products: bool = False) -> dict[str, Any]

Return the summary as JSON-compatible values; the product rows only when products is true.

spec_schema

spec_schema() -> dict[str, Any]

Return the JSON Schema of :class:RunSpec.

parse_spec

parse_spec(data: Mapping[str, Any]) -> RunSpec

Validate a run mapping, naming every offending field by its dotted path.

load_spec

load_spec(path: str | PathLike[str]) -> RunSpec

Read a .toml or .json run spec.

run async

run(
    spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
    *,
    runtimes: Mapping[str, RuntimeContext] | None = None,
    annotation_sources: Sequence[AnnotationSource] = (),
    connections: Mapping[str, Connection] | None = None,
) -> RunSummary

Ingest, score every model, annotate, and export, as the spec describes.

spec is a validated :class:RunSpec, a mapping to validate, or the path of a .toml or .json file. runtimes supplies an inline model's non-durable execution context by its spec name; annotation_sources adds sources built in code to those the spec declares, and their genome builds are checked against [run] genome_build as the declared ones are. connections replaces spec-declared connections by name with ones built in code; every name must be one the spec declares.

Progress, one line per stage that starts with the spec section it belongs to, such as [variants], goes to the altar.run logger at INFO, and the cohort's potential misses (:class:PotentialMiss) at WARNING when the run ends; altar run prints both on standard error.

The run closes every source and store it builds from the spec, once, when it ends: after the export, or when a check, a model or the export fails or the run is cancelled, in which case the original error propagates. Sources passed as annotation_sources and connections passed as connections stay open: the caller made them and closes them.

validate

validate(
    spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
    *,
    annotation_sources: Sequence[AnnotationSource] = (),
    connections: Mapping[str, Connection] | None = None,
) -> RunPlan

Check a spec as :func:run would, short of reading the cohort, and return what it would run.

It parses the spec, resolves every adapter type through the registries, builds the connections, the storage and every source, which may open a published copy over the network, and checks each model's and source's genome builds against [run] genome_build. It parses each model's configuration, run inputs and output selection, and builds the execution backend. A model that runs in containers needs a [backend], and the files a preparation step writes for it must be in the cache. When altar prepare published them, the model's binding checks that they match the model's configuration (ModelPlugin.prepared_run_inputs). Every annotation column a model's prioritization reads must come from a source of the spec or of annotation_sources. The backend's data_root must be a directory the run can create and write. With the local backend, each model's image is looked up on this machine and then in its registry; one that is in neither is a warning, since the network may be off. It checks that the cohort and the reference FASTA exist, but reads neither, and builds no store, so it creates no file. A default run directory that already holds files another run did not write is refused. Any problem raises :class:RunSpecError, as :func:run would; the arguments are :func:run's. It closes the sources it built before it returns or raises, as :func:run does.

spec_registries

spec_registries(
    spec: RunSpec | _PreparationSpec | _RegistrySpec,
    connections: Mapping[str, Connection] | None = None,
) -> Registries

Return the registries a spec can name: the installed ones, ALTAR_REGISTRIES's, then [registries].

connections holds the built connections a [registries] table names; without them a registry reads without one, which is enough to read the cache.

registries_of_spec

registries_of_spec(
    path: str | PathLike[str],
) -> tuple[Registries, Path]

Return the registries the spec at path can name, each with the connection it reads through, and its cache.

Only [run], [registries] and [connections] are read, so a spec still being written works. Raises RunSpecError for a spec that cannot be read or a section that does not parse.

entries_of_spec

entries_of_spec(
    path: str | PathLike[str],
    *,
    registries: Registries | None = None,
) -> list[EntryName]

Return every registry entry the spec at path names, each once and in order.

That is [variants] reference, then each [models.<name>] resources, a list's entries in order. Only those keys are read, so altar resources fetch --spec works on a spec still being written. registries are the ones the spec can name, so a reference whose first segment names none of them, such as grch38/ncbi/no-alt-analysis-set, is a genome; without them, the spec's own are read. Raises RunSpecError for an entry that does not parse.

fetch_command

fetch_command(
    spec: RunSpec | _PreparationSpec, spec_path: Path | None
) -> Callable[[EntryName, RegistrySource], str]

Return what builds the exact altar resources fetch command that fetches an entry a spec names.

A registry the spec's [registries] adds needs --spec, which also gives the cache; any other needs --cache-dir when the spec's cache is not the default one.

validate_preparation

validate_preparation(
    spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
    *,
    models: Sequence[str] | None = None,
    connections: Mapping[str, Connection] | None = None,
) -> dict[str, dict[str, Any]]

Check model preparation without hashing references, reading cohorts, or writing files (provisional).

A preparation-only spec may omit the cohort path and score store. A full run spec is also accepted: sources, stores, exports and interpretation sections are ignored. models selects models by key, or a resources list's section for all its models; by default every declared model must support preparation. Binding-owned configuration and training-assembly checks run.

prepare async

prepare(
    spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
    *,
    models: Sequence[str] | None = None,
    connections: Mapping[str, Connection] | None = None,
) -> PreparationSummary

Execute selected models' preparation workflows and collect their reusable artifacts (provisional).

Accepts the same sections as :func:validate_preparation. Every explicit invocation executes the selected workflows, even if outputs already exist. The reference is verified before plans execute. Binding-owned terminal files are collected into <cache_dir>/staging/<logical path>, where scoring reads them. No cohort, score store, source, or export is opened. Configuration and score lineage are unchanged, and a configured image is never replaced.

Outputs are first collected under preparation-attempts. Incomplete publication blocks subsequent scoring until preparation succeeds. A complete publication writes a record of the run inputs preparation ran with, and :func:run then lets each model's binding check the published files (ModelPlugin.prepared_run_inputs). Attempts are retained for diagnosis. Concurrent preparation and scoring against the same cache are unsupported. Preparation writes nothing to the run directory.

interpret async

interpret(
    spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
    *,
    sections: Sequence[str] | None = None,
    read_only: bool = False,
    read_variants: Sequence[str] | None = None,
    connections: Mapping[str, Connection] | None = None,
) -> InterpretationsSummary

Run the spec's [interpretations.<name>] sections, or read what they stored, and list the products.

Each section names a model of the spec, one method or several, a target and the variants: the run's cohort, ingested and normalized as run does, unless the section lists variants. Each method is one operation: :func:altar.interpretation.interpret prepares it, runs its missing variants on the spec's [backend], and stores every product in the [interpretation_store]. sections limits the run to those sections.

Each execution's label, its run_id, is <section>-<12 hex digits of the operation ID>, and the driver records its progress in <cache_dir>/interpretation-runs/<run_id>.json, beside the store, the files it stages and the files it collects. Running the same spec again therefore resumes an interrupted execution: one stopped while its tasks ran runs again, and one stopped after its files were collected is ingested from them. An execution that ended, with or without a failure, is never run again; a variant it left without products is run by a new execution, <run_id>-attempt2 and so on. A variant whose products are stored is not computed again.

With read_only nothing runs and no backend is needed: the stored products of each operation are read, limited to read_variants when given. The reference is still verified when its digest is asserted or a model reads it, because the requested operation identity is reconstructed from this spec.

The driver writes <work_dir>/interpretation-export/ in the run directory: products.tsv, one row per stored product file and variant (:class:InterpretationProduct); operations/<operation hex>.json, every operation the rows name, with its descriptors and input variants; and _interpretation_summary.json. A read limited to read_variants writes only the operations; its rows are in :attr:InterpretationsSummary.products.

An operation that fails, or leaves a variant without products, is logged and listed in :attr:InterpretationsSummary.failed, and the next operation still runs. A request refused before anything runs, such as a variant on a contig the model does not interpret, raises :class:RunSpecError naming the section.

Provisional: the function and its summary types are expected to change.

validate_interpretations

validate_interpretations(
    spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
    *,
    sections: Sequence[str] | None = None,
    connections: Mapping[str, Connection] | None = None,
) -> dict[str, dict[str, Any]]

Check the spec's interpretation sections as :func:interpret would, without reading or running anything.

Each section's model must be one that interprets, its methods and target ones the binding declares, and its configuration and run inputs must parse. The run needs a [backend] whose data_root it can write; with the local backend each runtime image is looked up, and one that is missing is a warning. Returns each section's model, plugin, methods, target and variants ("cohort" or the listed count). Provisional.