Run driver¶
altar.run is the Python form of the altar run, altar prepare and altar interpret commands. run() reads a
run spec, scores the cohort's cache misses with every model, annotates it with every source, and writes the export.
parse_spec() and load_spec() turn a mapping or a TOML or JSON file into a RunSpec.
Stable, except validate(), prepare(), interpret() and the summaries they return, which are provisional.
Provisional names lists them.
import asyncio
from altar.run import run
summary = asyncio.run(run("annotate.toml"))
print(summary.to_dict()["export_uri"])
from altar.run import interpret
summary = asyncio.run(interpret("interpret.toml"))
print(summary.products_uri, summary.failed)
The run spec reference lists every section and key, and Use Altar from Python
shows the driver in a program. altar spec-schema prints the JSON Schema of a spec.
run
¶
One-call reference driver: a declarative run spec resolved through the plugin registries.
A run is a mapping (TOML or JSON) that names a cohort, the models that score it and the annotation sources that
annotate it, and the adapters that execute and store their results. Every adapter axis resolves independently
through its entry-point registry by name, so a model, an execution backend, a score store, and an annotation
source are chosen without knowing each other (invariant 1). :func:run ingests the cohort, prepares and executes
every cache-miss batch of every model through :func:altar.scoring.score_cohort, and exports the materialized
results. The export reads each model's scores with the lineage resolution its preparation admitted the cohort with
(ScoreReusePolicy.exactly), so it reads the same lineages the cache checks did, including equivalent
lineages its binding declares, even when every variant was a hit and the run registered nothing. It is a
one-shot driver, not a scheduler: it retries nothing beyond what the engine does, and holds no queue.
A spec without models annotates: it ingests the cohort and exports every candidate variant with its sources'
columns, and needs no [backend]. It needs at least one source, declared or passed in code; a spec with neither
models nor sources is rejected. Its [store] is optional: without one the export reads through an in-memory
SQLite store, so the run writes no store file. A spec with models and no [store] keeps their scores in SQLite
at <cache_dir>/scores.sqlite, so later runs reuse them. Without models, a variant is prioritized only when
a source that prioritizes, such as AlphaMissense, flags it.
A run keeps its files in two directories ([run]). The run directory, work_dir, defaults to ./<job_id>
and holds what only this run reads: the candidate file, the files staged for and collected from this job's
containers (staging/jobs/<job_id>/ and results/jobs/<job_id>/), and the export. Deleting it loses no
reusable work. The cache, cache_dir, defaults to $ALTAR_CACHE_DIR or ./.altar and holds what a later
run reads: the files altar prepare publishes under staging/, the default score store, the default artifact
and interpretation stores, interpretation checkpoints, and the local backend's default data_root,
backend-data. It is created only when something is written there. A container file whose logical path starts
with jobs/ goes to the run directory, and every other one to the cache. A default run directory that already
exists, holds files, and is not an earlier run's is refused, so a job_id never writes into an unrelated
directory.
The run normalizes every variant it ingests, from a VCF or a canonical TSV, into
<work_dir>/variants/<job_id>.candidates.tsv. With [variants] reference_fasta each variant is left-aligned
against it (:func:altar.variants.left_align), so a deletion written at the right end of a repeat matches the
form gnomAD and ClinVar store; a variant on a contig the FASTA lacks, or whose REF the reference does not hold, is
only trimmed, and a model's preparation sets it aside. Without a reference each variant is trimmed to its minimal
representation (:func:altar_identity.minimal_representation). Two rows that normalize to one variant are one
candidate. A TSV's fields are separated by tabs or spaces, and its optional label is not carried over. The rows
a lookup may miss are counted, by reason, as :class:PotentialMiss entries of :attr:RunSummary.potential_misses
(provisional): indels only trimmed for want of a reference, which gnomAD and ClinVar key left-aligned; with a
reference, variants whose REF it does not hold or whose contig it lacks; and, in a run with sources, variants on
contigs other than chr1-22, X, Y and M, which the annotation datasets do not cover. They are advisory: the run
still exports every variant.
[run] genome_build is the assembly of the cohort's coordinates, normalized to Altar's spelling (GRCh38 is
read as hg38; see altar.models.canonical_genome_build). Every model scores the cohort on that assembly,
whatever assembly the model was trained on. Before any source is built, each declared source's class is checked
against it: a class whose supported_genome_builds leave the run's build out cannot serve it with any options,
so the run stops and names every such source, advising to remove it or lift the cohort over. Each built source's
identity().genome_build is then checked too, so an hg19 cohort is never annotated with hg38 data; only a source
whose class supports the run's build is advised to point at its data on that build. The UCSC and GRC spellings of
one assembly compare equal. A source that declares no identity is not checked here; the export treats it as
serving hg38, so it annotates no cohort on any other assembly.
[run] genome_build is checked against each model's run inputs the same way, before any source is built: a model
whose supported_genome_builds leave the run's build out is named in the same error. Then each model's
configuration, run inputs and output selection are parsed, and a binding's validation error becomes one line per
problem that names the spec setting to change. A model whose plan runs in containers needs a [backend], and
the files its plan reads from the platform, which a preparation step writes into <cache_dir>/staging, must be
there. A [backend] whose data_root names a directory the run cannot create or write is refused. With the
local backend each model's image is looked up with docker image inspect, then docker manifest inspect;
an image in neither place is a warning. All of this runs before the cohort is read.
A spec declares one [backend], or several [backends.<name>] in place of it. A container model runs on the
spec's only backend, or on the one its backend names, which it must name when there are several; its
preparation and interpretation run there too. Its pool replaces the GPU pool its binding's tasks ask for, for
every stage or per stage (score, preprocess, interpret), and leaves a task that asks for no GPU on the
CPU. A pool must be one the backend defines (ExecutionBackend.gpu_pools), and a binding's own pool too unless the
backend is cpu_only; both are checked before anything is submitted. Neither choice is part of the model's
identity, so stored scores are reused wherever they were computed; :attr:ModelRunSummary.compute and the
provenance record where this run computed. Models on different backends score one after another.
A VCF is read one line at a time. An ALT that names no sequence a variant key can hold (., *, a symbolic
<DEL>, a breakend, ALT equal to REF, or an IUPAC code) is skipped and counted by reason in
:attr:RunSummary.dropped_alleles. A malformed row, a VCF that would be read as a TSV, and a position that is not
ASCII digits are :class:RunSpecError, naming the row. Hashing a large reference FASTA is logged with its digest.
A supplied [variants] reference_digest is verified against the file before normalization or cache lookup. The
first check hashes the file and writes a verification record beside it; later runs accept the unchanged file from
that record without reading it.
[models.<name>] resources names a registry entry, "<registry>/<id>" such as "legnet/wtc11", optionally
pinned as "<registry>/<id>@sha256:<hex>" (see altar.registries). The entry gives the model's type,
which the section may repeat, and its saved configuration, which configuration may extend or override field by
field. [variants] reference names a genome of the genome registry by its id, such as
"grch38/ncbi/no-alt-analysis-set", or an alias, such as "hg38", or an entry of another registry as
"<registry>/<id>", instead of a path; a name whose first segment is no registry's is a genome's id.
[registries] adds registries to the installed ones and those of ALTAR_REGISTRIES: a name and its index's
URI, or {uri, connection, allow_private_hosts}.
A registry hosted elsewhere is read over https://, or from gs://, s3:// or hf:// anonymously
unless it names a connection, and may name only such addresses on public hosts; allow_private_hosts, on the
table or on its [connections.<name>], admits an intranet one (altar.registries). Every name resolves in the
cache, [run] cache_dir, to the files altar resources fetch put there; a run never downloads, and an
entry not fetched stops the run, and validate, with the command that fetches it. Before the binding builds
its configuration, each file's uri is replaced with its path in the cache. Identity reads digests only, so
the scores are those of the same configuration read from anywhere. A named genome's digest is its entry's, the
verification record the fetch wrote accepts the FASTA without hashing it, and it must be on [run]
genome_build. An entry's prepared files are put where altar prepare would have published them, with its
record, so the model scores without altar prepare; its binding checks them as it checks those
(ModelPlugin.prepared_run_inputs). The registry, its URI, the entry's id and its digest are recorded in
ModelRunSummary.registry and the provenance, never in identity.
resources may also be a list of entries, which the section expands into one model per entry, in order. Each
model's key is the entry's EntryName.model_key, <registry>-<id> in lowercase with . as _, such as
lab-encsr296uhq: the key a one-entry section [models.lab-encsr296uhq] resources = "lab/ENCSR296UHQ" has,
so the two forms reuse each other's scores, and moving an entry between sections changes nothing stored. The
section's other keys apply to every model; a type must be every entry's, model_name names one model and is
refused, and an entry named twice, or whose key is another section's, is refused. The section's key names it in
logs, ModelSectionSummary, ModelRunSummary.section and the provenance, never in identity. An entry whose
training_reference is not the run's reference is warned about, once per section with the counts.
Each model whose plugin reads the reference genome gets the FASTA, identified by its digest, as its run input
genome, and that digest is part of its scores' identity. A model that reads none
(ModelPlugin.reads_reference_genome), such as AlphaGenome, gets no reference, so adding or
changing reference_fasta leaves its identity alone. The FASTA is verified when a model reads it or the spec
asserts reference_digest, including annotation-only runs. Static validation does not read it.
The run logs one line per stage to the altar.run logger, starting with the spec section it belongs to, such as
[variants]: the cohort read, each source opened (with its release and where it reads), each model scored, and the
export written, with counts and elapsed times. A model's line counts the variants it set aside by reason and the
rows its store added. When the run ends it logs the skipped VCF alleles and the potential misses as warnings.
:func:validate (provisional) runs every check that needs no variant, building the sources and the backend but
reading no cohort and writing no file, and returns the :class:RunPlan it would run.
The :class:RunSummary records each source's identity, the release and build it served, as a
:class:SourceRunSummary (provisional, listed in __provisional__). A local export directory holds the summary
as _run_summary.json beside the exported files; the leading underscore keeps a Parquet reader that opens the
whole directory from taking it for a data file.
:attr:RunSummary.provenance records, for each model, the identity the run requested and the lineages its exported
rows came from, and the release and build of each source the export joins. The export carries the same JSON: a
Parquet file in its key-value metadata under altar.provenance, and a TSV in _provenance.json beside it.
A wide export, the default, also has _columns.tsv beside its table, the column manifest: one row per column,
naming the model, plugin, result schema and field each model column holds (altar.results.wide_column_manifest).
[export] layout = "long" (provisional) writes a models table, a variants table and one score table per
result schema in place of the wide table, with _provenance.json beside them in either format
(altar.results.render_long_export). Every layout reads the store's result buckets (ScoreStore.result_buckets),
so its rows are in genomic order: chromosomes 1, 2, …, X, Y, M, then other contigs, each by position.
Sections, with run and variants required, and at least one of models and sources::
[run] job_id, genome_build, work_dir (default ./<job_id>), cache_dir (default $ALTAR_CACHE_DIR or
./.altar)
[variants] path (a VCF or a canonical headerless TSV), format, reference | reference_fasta,
reference_digest (with reference_fasta only)
[registries] <name> = "<uri>" or {uri, connection}: registries beside the installed ones
[models.<name>] type, resources (an entry, or a list) | resource_manifest, configuration, run_inputs, output,
max_variants_per_batch, model_name, backend, pool (none: annotate only)
[backend] type, options, poll_interval_s, timeout, cpu_only (needed by container plans only)
[backends.<name>] the keys of [backend], in place of it: named backends each model chooses from
[connections.<name>] type, options
[storage] type, options, connection (default: local)
[store] type, options, connection (default: sqlite at <cache_dir>/scores.sqlite with models, in
memory without; a new sqlite store keeps lineages unless options
say lineages = false)
[detail_store] type, options, connection (default: the store's type on the store's connection, built
only for a model that declares detail tables; a store with a table_id
keeps them in <table_id>__details, a sqlite store at scores.sqlite keeps
them in scores.details.sqlite beside it, and an in-memory one in memory)
[artifact_store] type, options (required when a model's output selects an artifact; built only then)
[sources.<name>] type, options, connection
[export] format (parquet | tsv; default parquet; csv is a deprecated name for tsv), layout (wide |
long; default wide), destination, prioritized_only, bucket_rows (variants the store hands
the writers at a time; default: about 128 MiB of results)
Relative paths, both the spec's own and those inside adapter options and model configuration, which
Altar passes through verbatim, are resolved against the working directory the run is started from.
An adapter section's options are the keyword arguments of the registered class's constructor, so a
section can declare only an adapter whose constructor takes values a mapping can carry. The shipped
execution backends and score and detail stores do: local (data_root, slots, container_user)
and sqlite (path, and lineages for the score store). A store section that names a connection
is built through the store's from_options instead, which builds its client from the connection: the
bigquery score store takes table_id, lineage_table_id and primary_stage_rows, and the
bigquery detail store takes table_id, each reaching BigQuery through connection.bigquery_client()
and closing that client when the run ends. An annotation source is built through its
AnnotationSource.from_options hook instead, since most take a backend and a release record: the shipped
regions source takes data_dir, and the AlphaMissense, SpliceAI, GPN-Star, population-frequency, and
ClinVar bindings take dataset, the path of a dataset their builder wrote, whose metadata names the release
and genome build, plus the source's own settings. The population-frequency and ClinVar sources read their
publisher's copy of the latest release when no dataset is given, or a BigQuery table through a
connection, which a bigquery store joins into its own SQL. A source without the hook, such as the core
sqlite source, which takes AnnotationColumn values, and the bigquery source adapter, which takes a
client, is built in code and passed to :func:run as annotation_sources; a sources section that names one
fails before any scoring starts. Likewise an inline model whose execution needs an injected client takes it through
runtimes, keyed by the model's spec name; a spec is durable and never carries a credential or a client.
A [connections.<name>] section declares one way of reaching object storage or a warehouse, a registered
:class:~altar.connections.Connection built from its options: gcp (Application Default Credentials, with
project, anonymous, impersonate_service_account and requester_pays) or fsspec
(storage_options and protocols). A sources section, [storage], [store] or [detail_store]
names one with connection, and the driver passes the live object to it as the connection keyword: to the
source's or the store's from_options, or to the storage adapter's constructor. Only an adapter that names a
connection parameter takes one; a section that gives connection to any other adapter, or names a
connection the spec does not declare, is rejected, and a spec never names a client or a credential. Two
connections let one run read a dataset where its publisher hosts it, anonymously (public, for a source whose
from_options takes a connection), and keep its own files in the user's bucket, billed to their project and
read as their service account (lab)::
[connections.public]
type = "gcp"
options = { anonymous = true }
[connections.lab]
type = "gcp"
[connections.lab.options]
project = "my-lab-project"
impersonate_service_account = "reader@my-lab-project.iam.gserviceaccount.com"
[storage]
type = "fsspec"
connection = "lab"
The same lab connection keeps scores and details in BigQuery, billed to its project. A BigQueryScoreStore
can only join sources that read BigQuery in its SQL (SqlAnnotationSource), such as a population-frequency
source over an owned table. The run writes either export format itself, from the store's result buckets::
[store]
type = "bigquery"
connection = "lab"
options = { table_id = "my-lab-project.altar.scores" }
[detail_store]
type = "bigquery"
connection = "lab"
options = { table_id = "my-lab-project.altar.details" }
[export]
format = "parquet"
Without the [detail_store] section, the details go to my-lab-project.altar.scores__details through the
store's lab connection.
A host supplies a connection built in code, such as one holding a secret credential, through run(...,
connections={"lab": ...}). Every connection a run uses is declared in the spec, and the host's object wins:
it replaces the spec's declaration of the same name, which is then not built, so the spec still runs on its own
with its declared, ambient-credential default. Injecting a name the spec does not declare is an error rather
than a silent addition. A connection is operational: it never enters a source's, a model's, or a cache's
identity, so reading the same data through another connection reuses every score.
The [models.<name>] key is the model's stable model_id and the column prefix of its export block, so
keep it the same across runs of the same model instance for its scores to be cache hits.
A spec with models that keeps its scores in a new SQLite store, in memory or at a path that holds no scores yet,
gets lineages = true unless its options set lineages. A model whose setting changes then gets a lineage of
its own beside the old one. A store file that already holds scores is opened in the mode it was created with and
never converted; a lineages option that disagrees with it is refused. In a store without lineages, a changed
setting stops the run with a :class:RunSpecError that names both ways on: a new [models.<name>] key, or a new
store with lineages = true.
A model's output is its :class:~altar.models.OutputSelection, and naming an artifact the binding declares
in its outputs (Sei's chromatin_profile_effects, for example) makes the run produce and store it. Such a
run needs an [artifact_store]; a spec that selects an artifact without one is rejected before anything is
ingested. The shipped storage store copies files through the run's [storage] and is built with it; its
root_uri defaults to <cache_dir>/artifacts and its SQLite index_path to
<cache_dir>/artifacts/index.sqlite, so a later run's artifacts are cache hits too::
[models.sei.output]
outputs = ["scores", "sequence_class_effects", "chromatin_profile_effects"]
reducer = "official_nucleosome_adjusted_sequence_class_effect_v1"
[artifact_store]
type = "storage"
EntryText
module-attribute
¶
A registry entry as a spec writes it, <registry>/<id> or <registry>/<id>@sha256:<hex>.
BackendName
module-attribute
¶
A [backends.<name>] key: a simple identifier, which may also hold hyphens.
RegistryValue
module-attribute
¶
RegistryValue = (
Annotated[str, Field(min_length=1)] | RegistrySettings
)
A registry's index URI, or {uri, connection, allow_private_hosts}.
RunSpecError
¶
Bases: ValueError
The run spec is malformed, or names something the installed registries cannot build.
RunSettings
¶
Bases: _SpecModel
Identity and directories of one run.
work_dir is the run directory: the candidate file, this job's container files and the export, which no
later run reads. Without it the run directory is ./<job_id> (:attr:run_dir). cache_dir holds
everything a later run reuses: prepared model files, the default score, artifact and interpretation stores,
interpretation checkpoints, and the local backend's default data_root. Without it, it is
$ALTAR_CACHE_DIR, or ./.altar when that is not set, read when the spec is parsed.
run_dir
property
¶
Return the run directory: work_dir, or ./<job_id> when the spec leaves it out.
VariantSettings
¶
Bases: _ReferenceSettings
The cohort: a VCF or canonical TSV, and the reference its models score against.
ModelSettings
¶
Bases: _SpecModel
One model instance: its registry type, durable configuration, and per-run inputs.
type
class-attribute
instance-attribute
¶
The registered model type, such as LEGNET; a registry entry named with resources gives it.
configuration
class-attribute
instance-attribute
¶
The binding's configuration fields; with resources or resource_manifest, the ones to add or change.
resources
class-attribute
instance-attribute
¶
A registry entry, <registry>/<id> such as legnet/wtc11, fetched with altar resources fetch, or a
list of them, which the section expands into one model per entry.
It is an alternative to resource_manifest: the entry gives the model's type and saved configuration. It may
be pinned to one entry document as <registry>/<id>@sha256:<hex>. A list's models are keyed by each entry's
EntryName.model_key, such as lab-encsr296uhq, and share the section's other keys.
resource_manifest
class-attribute
instance-attribute
¶
A binding-written resource manifest; configuration then holds the remaining settings.
registry_entry
property
¶
registry_entry: FetchedEntry | None
The registry entry resources named, once the run resolved it in the cache; None before or without.
section
property
¶
The [models.<section>] whose resources list this model was expanded from; None for its own.
entries
property
¶
entries: tuple[EntryName, ...]
The entries resources names, parsed: one for a string, each of a list, none without.
run_inputs
class-attribute
instance-attribute
¶
Binding run inputs beyond genome_build and genome, which the run supplies.
backend
class-attribute
instance-attribute
¶
backend: BackendName | None = None
The [backends.<name>] the model's containers run on: scoring, altar prepare and altar interpret.
It is required only when the spec declares several backends; with one, every model runs on it. Where a model runs is not part of its identity, so its stored scores are reused whichever backend computed them.
pool
class-attribute
instance-attribute
¶
The GPU pool the model's GPU tasks ask for, in place of the binding's default; a name its backend defines.
A table sets it per stage, by the operations the binding declares (score, preprocess, interpret):
pool = {score = "a100"}. A stage the table leaves out keeps the binding's pool, and a task that asks for no
GPU stays on the CPU. Like backend, it is not part of the model's identity.
RegistrySettings
¶
Bases: _SpecModel
A [registries] value given as a table: the index's URI, the connection that reads it, and its opt-in.
allow_private_hosts lets a registry hosted elsewhere name hosts on a private network, such as an intranet
mirror; without it, a fetch refuses any host that resolves to a private, loopback or link-local address.
AdapterSettings
¶
Bases: _SpecModel
A registered adapter and the keyword arguments of its constructor.
ConnectionSettings
¶
Bases: AdapterSettings
A [connections.<name>] section: its connection type, the type's options, and its private-host opt-in.
allow_private_hosts lets every registry hosted elsewhere that reads through this connection name hosts on
a private network, as a [registries] table's own key does. It is not passed to the connection.
ConnectedAdapterSettings
¶
Bases: AdapterSettings
An adapter that may read through one of the spec's named connections.
connection
class-attribute
instance-attribute
¶
A [connections.<name>] key; the live connection is passed to the adapter as connection.
BackendSettings
¶
Bases: AdapterSettings
The execution backend, how often the run checks its tasks, and how long it waits for them.
timeout
class-attribute
instance-attribute
¶
timeout: Annotated[
float,
Field(gt=0),
WithJsonSchema(
{
"anyOf": [
{"type": "number", "exclusiveMinimum": 0},
{
"type": "string",
"pattern": "^\\s*\\d+(\\.\\d+)?\\s*[smhdSMHD]\\s*$",
},
],
"default": 86400,
}
),
] = 86400.0
How long the run waits on one set of tasks, in seconds: one model's batch, or one wave of workflow stages
for altar prepare and altar interpret. A spec may write it as text such as "90m", "24h" or
"2d". Tasks still open after it are cancelled, where the backend can cancel a task, and the run stops.
cpu_only
class-attribute
instance-attribute
¶
Clear every task's GPU request so a plan built for accelerators runs on a CPU-only host.
ExportSettings
¶
Bases: _SpecModel
The export written after every model has scored.
format is parquet (the default), which keeps repeated and struct columns as lists and structs, or
tsv, gzipped tab-separated text with those columns as JSON text. csv is a deprecated name for tsv:
the file was always tab-separated, and a run that names it logs a warning.
layout is wide (the default), one table with a row per variant and a block of columns per model, with
its column manifest _columns.tsv beside it, or long (provisional), a models table, a variants
table and one score table per result schema with a row per variant and model (see
altar.results.write_long_export).
bucket_rows
class-attribute
instance-attribute
¶
Variants per result bucket the export reads from the store at a time; by default a size of about 128 MiB.
InterpretationSettings
¶
Bases: _SpecModel
One interpretation request: a model of the spec, the method to run, the target it explains, and the variants.
model names a [models.<name>] section, whose type, configuration, resource manifest and run inputs the
interpretation uses; that key is also the model ID its products are stored under. method names one method
and methods several, each run as its own operation; a method runs every method it requires. target
defaults to the methods' only target. variants lists canonical variant IDs to interpret instead of the
run's cohort. configuration and run_inputs are added to the model's own, for settings only
interpretation reads, such as ChromBPNet's motifs.
requested_methods
property
¶
Return the methods the section asks for, in order.
InterpretationStoreSettings
¶
Bases: _SpecModel
Where interpretation products are kept: the reference store, a file tree and its SQLite index.
options takes root_uri, the tree the product files are copied to through the run's [storage]
(default <cache_dir>/interpretations), and index_path, the SQLite index (default index.sqlite in
that directory). storage is the only type until an interpretation-store entry-point group exists.
RunSpec
¶
Bases: _SpecModel
A complete, validated run.
models
class-attribute
instance-attribute
¶
models: dict[Name, ModelSettings] = {}
The models to score; a spec without any annotates the cohort through its sources and exports those.
backend
class-attribute
instance-attribute
¶
backend: BackendSettings | None = None
The run's one backend, which every model's containers run on.
backends
class-attribute
instance-attribute
¶
backends: dict[BackendName, BackendSettings] = {}
Named backends, in place of backend; each model names the one it runs on with backend.
connections
class-attribute
instance-attribute
¶
connections: dict[Name, ConnectionSettings] = {}
Named ways of reaching object storage and warehouses, which storage and sources refer to.
registries
class-attribute
instance-attribute
¶
registries: dict[Name, RegistryValue] = {}
Registries of models and genomes beside the installed ones, by name: an index's URI or a RegistrySettings.
store
class-attribute
instance-attribute
¶
store: ConnectedAdapterSettings | None = None
Where scores are kept, and the store the export reads through. A spec with models that leaves it out keeps
them in SQLite at <cache_dir>/scores.sqlite; one without models gets an in-memory SQLite store, so the run
writes no store file.
artifact_store
class-attribute
instance-attribute
¶
artifact_store: AdapterSettings | None = None
Where the artifacts a model's output selects are stored; required when any model selects one.
interpretations
class-attribute
instance-attribute
¶
interpretations: dict[Name, InterpretationSettings] = {}
Interpretation requests, which altar interpret runs (provisional); altar run leaves them alone.
interpretation_store
class-attribute
instance-attribute
¶
interpretation_store: InterpretationStoreSettings | None = (
None
)
Where altar interpret keeps interpretation products (provisional); defaults to the reference store.
ModelRunSummary
dataclass
¶
ModelRunSummary(
name: str,
plugin_id: str,
candidate_count: int,
duplicate_count: int,
ineligible_count: int,
cache_hit_count: int,
scored_count: int,
batch_count: int,
added: int,
skipped: int,
ineligible_report: str | None,
details: tuple[DetailRunResult, ...] = (),
artifacts: tuple[ArtifactRunResult, ...] = (),
variants_without_rows: int = 0,
ineligible_reasons: dict[str, int] = dict(),
compute: dict[str, Any] | None = None,
registry: dict[str, str] | None = None,
section: str | None = None,
)
Preparation counts and persisted rows for one model of the run.
variants_without_rows
class-attribute
instance-attribute
¶
Variants sent to the model that got no primary row, summed over its batches.
ineligible_reasons
class-attribute
instance-attribute
¶
ineligible_count by reason, from the ineligible-variant report: an IneligibilityReason value, or for
unsupported_variant_class the variant's class, as unsupported_variant_class:deletion.
compute
class-attribute
instance-attribute
¶
Where the model's containers ran, or None for a model that runs inline (provisional): backend, the
[backends.<name>] key (None for the spec's single [backend]), its registered type, pools, the
GPU pools its scoring tasks ask for after the model's pool, and cpu_only, whether the backend dropped
those GPU requests. It records where this run computed, not where cached scores came from.
registry
class-attribute
instance-attribute
¶
The registry entry resources named (provisional): registry, its name; uri, where its index is;
id, the entry's id; and entry_sha256, the entry document's digest. None for a model named another
way. Like compute, it is not part of the model's identity.
section
class-attribute
instance-attribute
¶
The [models.<section>] whose resources list the model was expanded from (provisional); None for a
model with a section of its own. Not part of the model's identity: its key, name, is the entry's.
ModelSectionSummary
dataclass
¶
SourceRunSummary
dataclass
¶
One annotation source the run exported: the release and genome build it served, as it declares them.
PotentialMiss
dataclass
¶
Variants of the cohort that a lookup may miss or read at the wrong place; advisory, they are still exported.
reason is one of:
indel_not_left_aligned: an indel only trimmed, because the spec gives no[variants] reference_fastato left-align it against; gnomAD and ClinVar key indels left-aligned, so one written further right in a repeat is not found;reference_mismatch: REF is not the reference's bases at that position, so the variant was only trimmed, and a lookup reads it as written (only checked with a reference);contig_not_in_reference: the reference FASTA has no contig of that name (only checked with a reference);contig_not_primary: a contig other than chr1-22, X, Y and M, such as an alt or unplaced contig, which the annotation datasets do not cover, so its source columns are empty (only counted when the run has annotation sources).
RunSummary
dataclass
¶
RunSummary(
job_id: str,
genome_build: str,
candidate_variants: str,
variant_count: int,
models: tuple[ModelRunSummary, ...],
export_uri: str,
sources: tuple[SourceRunSummary, ...] = (),
potential_misses: tuple[PotentialMiss, ...] = (),
dropped_alleles: dict[str, int] = dict(),
provenance: dict[str, Any] = dict(),
sections: tuple[ModelSectionSummary, ...] = (),
export_layout: str = "wide",
column_manifest_uri: str | None = None,
export_tables: dict[str, str] = dict(),
)
What :func:run ingested, scored, annotated, and exported.
potential_misses
class-attribute
instance-attribute
¶
potential_misses: tuple[PotentialMiss, ...] = ()
Variants a lookup may miss, by reason (see :class:PotentialMiss); empty when none was found.
dropped_alleles
class-attribute
instance-attribute
¶
VCF ALT alleles that name no sequence a variant key can hold, by reason, so they are not in the export:
no_alternate (.), spanning_deletion (*), symbolic (<DEL>), breakend,
reference_only (ALT equal to REF), and not_acgtn (an IUPAC code or another base outside ACGTN).
provenance
class-attribute
instance-attribute
¶
What produced the export's model scores and source columns, as JSON-compatible values.
The export carries the same payload. It is {"format": 1, "models": {<spec name>: {...}}, "sources": {<source
name>: {...}}}. Each source the export joins maps to its identity, as :class:SourceRunSummary records it
(source_id, schema_version, release and genome_build), or to null for a source that declares
none. Each model's entry holds:
model_name,column_prefix(the prefix of its export columns) andplugin_id;requested: the identity this run asked for, which it stamps on the rows it writes. It holds thelineage_id,plugin_version,configuration_hash,scientific_configuration_hash,reducer,output_selection_hash,result_schema_hash,runtimes(eachname,kindandidentity, such as a container image digest),genome_build,outputsandresult_schema_version;reuse: the lineages the run read.scopeis the widest step,current,equivalentoraccepted.lineageslists eachlineage_idandstepin preference order, or isnullfor a store that keeps no lineages.equivalence_rules_idnames the binding's equivalence rules when they ran;this_run:cache_hits, the variants the run found already scored, androws_added, the rows it wrote;compute: where this run ran the model's containers, as :attr:ModelRunSummary.computerecords it (backend,type,poolsandcpu_only), ornullfor an inline model. It is not part of the model's identity: a cache hit's producer may have run anywhere;registry: the registry entry that named the model, asModelRunSummary.registryrecords it, andsection: the section whoseresourceslist it was expanded from, ornullfor either. Neither is part of the model's identity;producers: one entry per lineage the exported rows came from, in preference order. Each has itslineage_id,step,rows(the exported rows under it) andrecorded_runs, the identities the store recorded as writing under it, empty for an imported lineage. The store does not record which of them wrote a given row. A store that keeps no lineages gives"unknown".
A cache hit is never attributed to this run: its producer is the lineage it was stored under.
export says how the export is laid out: layout, wide or long, and column_manifest, the
file beside a wide export that describes its columns, _columns.tsv, or null for a long export.
sections
class-attribute
instance-attribute
¶
sections: tuple[ModelSectionSummary, ...] = ()
Each section whose resources list expanded into several of models, with its count (provisional).
export_layout
class-attribute
instance-attribute
¶
wide or long, as [export] layout set it.
column_manifest_uri
class-attribute
instance-attribute
¶
Where a wide export's column manifest, _columns.tsv, was written: one row per column of the table, naming
the model, plugin, result schema and field of each model column (provisional). None for a long export.
export_tables
class-attribute
instance-attribute
¶
A long export's tables, each table's path within the export (models, variants, scores/<name>) to
the directory it was written to (provisional). Empty for a wide export.
RunPlan
dataclass
¶
RunPlan(
job_id: str,
genome_build: str,
work_dir: str,
cache_dir: str,
variants: str,
reference_fasta: str | None,
models: dict[str, str],
sources: tuple[SourceRunSummary, ...],
store: str,
export_format: str,
export_uri: str,
compute: dict[str, dict[str, Any] | None] = dict(),
sections: tuple[ModelSectionSummary, ...] = (),
export_layout: str = "wide",
)
What a valid spec would run, from :func:validate: nothing was ingested, scored, or written.
work_dir
instance-attribute
¶
The run directory: [run] work_dir, or <job_id> when the spec leaves it out.
cache_dir
instance-attribute
¶
The cache, [run] cache_dir, where reusable files are kept; a run creates it only to write there.
models
instance-attribute
¶
Each model's spec name and the plugin_id of the binding that scores it.
sources
instance-attribute
¶
sources: tuple[SourceRunSummary, ...]
Every annotation source, built and checked against genome_build, with the release it serves.
store
instance-attribute
¶
The score store's registered type; for a spec without [store], sqlite (<cache_dir>/scores.sqlite)
with models and sqlite (in memory) without.
export_format
instance-attribute
¶
parquet or tsv; a spec's deprecated csv is reported as tsv.
compute
class-attribute
instance-attribute
¶
Each model's backend and GPU pools, as :attr:ModelRunSummary.compute records them; None for an inline
model (provisional).
sections
class-attribute
instance-attribute
¶
sections: tuple[ModelSectionSummary, ...] = ()
Each section whose resources list expanded into several of models, with its count (provisional).
export_layout
class-attribute
instance-attribute
¶
wide or long, as [export] layout sets it.
ModelPreparationSummary
dataclass
¶
ModelPreparationSummary(
name: str,
plugin_id: str,
completed_stages: tuple[str, ...],
output_uris: tuple[str, ...],
compute: dict[str, Any] | None = None,
)
One model's completed preparation stages and collected output locations (provisional).
compute
class-attribute
instance-attribute
¶
The backend preparation ran on and the GPU pools its tasks asked for, as ModelRunSummary.compute records
them.
PreparationSummary
dataclass
¶
PreparationSummary(
job_id: str, models: tuple[ModelPreparationSummary, ...]
)
The model artifacts prepared under a run's staging directory (provisional).
InterpretationProduct
dataclass
¶
InterpretationProduct(
section: str,
model_id: str,
method: str,
variant_id: str,
product: str,
shard: str,
run_id: str,
operation_id: str,
interpretation_id: str,
uri: str,
row: int,
chrom: str,
start: int,
end: int,
anchor: int,
media_type: str,
digest: str,
)
One stored product file that describes a variant: where its bytes are, which row, and what made it.
method is the requested method of the section's operation. operation_id and run_id name the
operation and execution whose commit made the file visible, which may be an earlier request's: its operation
file names its input variants. interpretation_id names the method run, and its descriptor in the operation
file names the model resources, runtimes and genome. row is the variant's row in the file, and chrom,
start, end and anchor give the 0-based, half-open reference span the row covers and the offset of
the variant's first base in it. shard holds the file's labels, such as allele=ref.
InterpretationOperationSummary
dataclass
¶
InterpretationOperationSummary(
name: str,
model_id: str,
plugin_id: str,
method: str,
target: str,
operation_id: str,
interpretation_ids: dict[str, str],
operation_uri: str,
variant_count: int,
cached_count: int,
executed_count: int,
run_id: str | None,
resumed: bool,
committed: tuple[str, ...],
skipped: dict[str, str],
failure: str | None,
missing: tuple[str, ...],
product_files: int,
)
One operation interpret ran or read: one method of an [interpretations.<name>] section.
operation_id names the request: the method chain's interpretation IDs, by method in
interpretation_ids, and the input variants. operation_uri is its JSON file, with every descriptor and
the variants. cached_count counts the variants whose products were stored before, and executed_count
those the execution run_id ran; run_id is None when nothing ran. committed and skipped
say which methods that execution stored, failure is the workflow's error, and missing lists the
variants that still have no complete set of products.
complete
property
¶
Return whether every requested variant has every product of the method chain.
InterpretationsSummary
dataclass
¶
InterpretationsSummary(
job_id: str,
genome_build: str,
store: dict[str, str],
products_uri: str | None,
operations: tuple[InterpretationOperationSummary, ...],
products: tuple[InterpretationProduct, ...] = (),
)
What :func:interpret ran or read, and where it wrote the listing of the stored products.
products_uri is products.tsv, one row per :class:InterpretationProduct, or None for a read
limited to some variants, which writes no listing. operations holds one
:class:InterpretationOperationSummary per requested method of each section, and products the listed
rows. store names the interpretation store's root_uri and index_path.
parse_spec
¶
parse_spec(data: Mapping[str, Any]) -> RunSpec
Validate a run mapping, naming every offending field by its dotted path.
run
async
¶
run(
spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
*,
runtimes: Mapping[str, RuntimeContext] | None = None,
annotation_sources: Sequence[AnnotationSource] = (),
connections: Mapping[str, Connection] | None = None,
) -> RunSummary
Ingest, score every model, annotate, and export, as the spec describes.
spec is a validated :class:RunSpec, a mapping to validate, or the path of a .toml or .json
file. runtimes supplies an inline model's non-durable execution context by its spec name;
annotation_sources adds sources built in code to those the spec declares, and their genome builds are
checked against [run] genome_build as the declared ones are. connections replaces spec-declared
connections by name with ones built in code; every name must be one the spec declares.
Progress, one line per stage that starts with the spec section it belongs to, such as [variants], goes to
the altar.run logger at INFO, and the cohort's potential misses (:class:PotentialMiss) at WARNING
when the run ends; altar run prints both on standard error.
The run closes every source and store it builds from the spec, once, when it ends: after the export, or when
a check, a model or the export fails or the run is cancelled, in which case the original error propagates.
Sources passed as annotation_sources and connections passed as connections stay open: the caller made
them and closes them.
validate
¶
validate(
spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
*,
annotation_sources: Sequence[AnnotationSource] = (),
connections: Mapping[str, Connection] | None = None,
) -> RunPlan
Check a spec as :func:run would, short of reading the cohort, and return what it would run.
It parses the spec, resolves every adapter type through the registries, builds the connections, the storage
and every source, which may open a published copy over the network, and checks each model's and source's
genome builds against [run] genome_build. It parses each model's configuration, run inputs and output
selection, and builds the execution backend. A model that runs in containers needs a [backend], and the
files a preparation step writes for it must be in the cache. When altar prepare published
them, the model's binding checks that they match the model's configuration (ModelPlugin.prepared_run_inputs).
Every annotation column a model's prioritization reads must come from a source of the spec or of
annotation_sources. The backend's data_root must be a directory the run can create and write.
With the local backend, each model's image is looked up on
this machine and then in its registry; one that is in neither is a warning, since the network may be off.
It checks that the cohort and the reference FASTA exist, but reads neither, and builds no store, so it
creates no file. A default run directory that already holds files another run did not write is refused. Any
problem raises :class:RunSpecError, as :func:run would; the arguments are :func:run's. It closes the
sources it built before it returns or raises, as :func:run does.
spec_registries
¶
spec_registries(
spec: RunSpec | _PreparationSpec | _RegistrySpec,
connections: Mapping[str, Connection] | None = None,
) -> Registries
Return the registries a spec can name: the installed ones, ALTAR_REGISTRIES's, then [registries].
connections holds the built connections a [registries] table names; without them a registry reads
without one, which is enough to read the cache.
registries_of_spec
¶
registries_of_spec(
path: str | PathLike[str],
) -> tuple[Registries, Path]
Return the registries the spec at path can name, each with the connection it reads through, and its cache.
Only [run], [registries] and [connections] are read, so a spec still being written works. Raises
RunSpecError for a spec that cannot be read or a section that does not parse.
entries_of_spec
¶
entries_of_spec(
path: str | PathLike[str],
*,
registries: Registries | None = None,
) -> list[EntryName]
Return every registry entry the spec at path names, each once and in order.
That is [variants] reference, then each [models.<name>] resources, a list's entries in order. Only those
keys are read, so altar resources fetch --spec works on a spec still being written. registries are the
ones the spec can name, so a reference whose first segment names none of them, such as
grch38/ncbi/no-alt-analysis-set, is a genome; without them, the spec's own are read. Raises RunSpecError
for an entry that does not parse.
fetch_command
¶
fetch_command(
spec: RunSpec | _PreparationSpec, spec_path: Path | None
) -> Callable[[EntryName, RegistrySource], str]
Return what builds the exact altar resources fetch command that fetches an entry a spec names.
A registry the spec's [registries] adds needs --spec, which also gives the cache; any other needs
--cache-dir when the spec's cache is not the default one.
validate_preparation
¶
validate_preparation(
spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
*,
models: Sequence[str] | None = None,
connections: Mapping[str, Connection] | None = None,
) -> dict[str, dict[str, Any]]
Check model preparation without hashing references, reading cohorts, or writing files (provisional).
A preparation-only spec may omit the cohort path and score store. A full run spec is also accepted: sources,
stores, exports and interpretation sections are ignored. models selects models by key, or a resources
list's section for all its models; by default every declared model must support preparation. Binding-owned
configuration and training-assembly checks run.
prepare
async
¶
prepare(
spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
*,
models: Sequence[str] | None = None,
connections: Mapping[str, Connection] | None = None,
) -> PreparationSummary
Execute selected models' preparation workflows and collect their reusable artifacts (provisional).
Accepts the same sections as :func:validate_preparation. Every explicit invocation executes the selected
workflows, even if outputs already exist. The reference is verified before plans execute. Binding-owned
terminal files are collected into <cache_dir>/staging/<logical path>, where scoring reads them. No cohort,
score store, source, or export is opened. Configuration and score lineage are unchanged, and a configured
image is never replaced.
Outputs are first collected under preparation-attempts. Incomplete publication blocks subsequent scoring
until preparation succeeds. A complete publication writes a record of the run inputs preparation ran with,
and :func:run then lets each model's binding check the published files
(ModelPlugin.prepared_run_inputs). Attempts are retained for diagnosis. Concurrent preparation and scoring
against the same cache are unsupported. Preparation writes nothing to the run directory.
interpret
async
¶
interpret(
spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
*,
sections: Sequence[str] | None = None,
read_only: bool = False,
read_variants: Sequence[str] | None = None,
connections: Mapping[str, Connection] | None = None,
) -> InterpretationsSummary
Run the spec's [interpretations.<name>] sections, or read what they stored, and list the products.
Each section names a model of the spec, one method or several, a target and the variants: the run's cohort,
ingested and normalized as run does, unless the section lists variants. Each method is one operation:
:func:altar.interpretation.interpret prepares it, runs its missing variants on the spec's [backend],
and stores every product in the [interpretation_store]. sections limits the run to those sections.
Each execution's label, its run_id, is <section>-<12 hex digits of the operation ID>, and the driver
records its progress in <cache_dir>/interpretation-runs/<run_id>.json, beside the store, the files it
stages and the files it collects. Running the same spec again
therefore resumes an interrupted execution: one stopped while its tasks ran runs again, and one stopped
after its files were collected is ingested from them. An execution that ended, with or without a failure, is
never run again; a variant it left without products is run by a new execution, <run_id>-attempt2 and so
on. A variant whose products are stored is not computed again.
With read_only nothing runs and no backend is needed: the stored products of each operation are read,
limited to read_variants when given. The reference is still verified when its digest is asserted or a
model reads it, because the requested operation identity is reconstructed from this spec.
The driver writes <work_dir>/interpretation-export/ in the run directory: products.tsv, one row per
stored product file
and variant (:class:InterpretationProduct); operations/<operation hex>.json, every operation the rows
name, with its descriptors and input variants; and _interpretation_summary.json. A read limited to
read_variants writes only the operations; its rows are in :attr:InterpretationsSummary.products.
An operation that fails, or leaves a variant without products, is logged and listed in
:attr:InterpretationsSummary.failed, and the next operation still runs. A request refused before anything
runs, such as a variant on a contig the model does not interpret, raises :class:RunSpecError naming the
section.
Provisional: the function and its summary types are expected to change.
validate_interpretations
¶
validate_interpretations(
spec: RunSpec | Mapping[str, Any] | str | PathLike[str],
*,
sections: Sequence[str] | None = None,
connections: Mapping[str, Connection] | None = None,
) -> dict[str, dict[str, Any]]
Check the spec's interpretation sections as :func:interpret would, without reading or running anything.
Each section's model must be one that interprets, its methods and target ones the binding declares, and its
configuration and run inputs must parse. The run needs a [backend] whose data_root it can write; with
the local backend each runtime image is looked up, and one that is missing is a warning. Returns each
section's model, plugin, methods, target and variants ("cohort" or the listed count). Provisional.