Interpretation¶
altar.interpretation runs interpretation operations, such as ChromBPNet's DeepSHAP contributions, and stores
their products. Bindings that declare interpretation methods import from it too. interpret() runs one request on
an execution backend. An application that drives execution itself calls prepare_interpretation(),
ingest_interpretation_results() and read_interpretation(), as
Run and read interpretations shows.
Provisional: every name in this module is provisional. See Provisional names.
interpretation
¶
Interpretation operations: explain a model's predictions for variants, and store what each method writes.
A binding opts in by subclassing :class:InterpretableModel and declaring :class:InterpretationMethod values.
:func:interpret runs a request in one call on an execution backend. A host that drives execution itself prepares
a request with :func:prepare_interpretation, runs the plan with :func:~altar.models.run_model_workflow, stores
the product files with :func:ingest_interpretation_results, and reads them back with :func:read_interpretation.
:class:StorageInterpretationStore is the reference :class:InterpretationStore.
Every name here is provisional and listed in __provisional__: it is expected to change.
Stored interpretation records are keyed by their interpretation ID.
InterpretationCommitError
¶
Bases: ValueError
A commit, or a put that a commit would rely on, breaks the store's rules; nothing was made visible.
InterpretationKey
dataclass
¶
InterpretationKey(
run_id: str,
shard: tuple[tuple[str, str], ...] = (),
variant_id: str | None = None,
)
The execution that wrote one product file, and the variant a per-variant file is for.
run_id labels one execution of an interpretation plan; the host chooses it. shard holds further
labels that tell a method's files of one product apart, such as ("allele", "ref"), as sorted pairs.
variant_id is set only for a variant-scoped product.
InterpretationRecord
dataclass
¶
InterpretationRecord(
model_id: str,
product: str,
interpretation_id: str,
key: InterpretationKey,
rows: tuple[InterpretationRow, ...],
media_type: str,
digest: str,
byte_size: int,
uri: str,
operation_id: str | None = None,
)
One stored product file: the method run and execution that wrote it, its rows, and where its bytes are.
operation_id is the operation whose commit made the record visible. A record :meth:InterpretationStore.put
returns is not committed yet, so its operation_id is None. rows follow the rows of the file in
order. digest, byte_size and uri are as on :class:~altar.results.ArtifactRecord.
InterpretationRow
dataclass
¶
The variant one row of a product file describes, and the reference span the row's data covers.
start and end are a 0-based, half-open interval on chrom. anchor is the offset of the
variant's first base in that interval, so start + anchor is the variant's 0-based position. A row of a
per-base array maps its element i to reference base start + i; the binding documents how bases after
an indel's alternate allele map.
InterpretationStore
¶
Bases: ABC
Persistence axis for interpretation products.
A store keeps the operations it was given, the descriptor and product schemas of each interpretation ID, the files put under them, and the commits that made files visible. Reads take the interpretation IDs whose records count, in preference order. The store never decodes a file.
Provisional: the contract is expected to change.
close
¶
Release the connections and clients this store opened itself; a second call does nothing.
record_operation
abstractmethod
async
¶
record_operation(
operation: InterpretationOperation,
) -> None
Record one operation, idempotently by its operation_id.
The store checks that every descriptor hashes to its interpretation ID and that the operation ID
matches its contents, and raises :class:~altar.interpretation.InterpretationIdentityError otherwise.
A second operation with the same ID keeps the first record.
read_operation
abstractmethod
async
¶
read_operation(
model_id: str, operation_id: str
) -> InterpretationOperation | None
Return the recorded operation, verified again, or None when the store has none.
ensure_product
abstractmethod
async
¶
ensure_product(
*,
model_id: str,
model_name: str,
descriptor: InterpretationDescriptor,
schema: ArtifactSchema,
) -> None
Declare one product under the descriptor's interpretation ID.
The descriptor is stored unhashed. A product already declared under the ID with another name, media
type or scope raises :class:~altar.results.SchemaCollisionError.
put
abstractmethod
async
¶
put(
model_id: str,
product: str,
*,
interpretation_id: str,
key: InterpretationKey,
rows: Sequence[InterpretationRow],
source_uri: str,
storage: Storage,
digest: str,
) -> InterpretationRecord
Copy the file at source_uri (readable through storage) into the store and index it, uncommitted.
The product must have been declared with :meth:ensure_product first, and key and rows must fit
its scope (:func:require_interpretation_rows). digest is the sha256: identity of the bytes,
which the store checks. A put of bytes the store already holds under the same key returns the existing
record. Different bytes under the same key are another record.
commit
abstractmethod
async
¶
commit(
model_id: str,
*,
operation_id: str,
run_id: str,
method: str,
records: Sequence[InterpretationRecord],
) -> tuple[InterpretationRecord, ...]
Make one method's records of one execution visible together, and return them as committed.
The operation must be recorded and list method. Every record must have been put under the method's
interpretation ID and keyed by run_id, with one record per product and key. Every product declared
under the ID must have records, and each product's rows must describe the same variants. A method with an
upstream method commits only after that method committed in the same execution, over the same variants.
An execution belongs to one operation. Committing the same records again returns them; committing other
records for the same operation, execution and method raises :class:InterpretationCommitError.
get_missing_variants
abstractmethod
async
¶
get_missing_variants(
model_id: str,
product: str,
variant_ids: Sequence[str],
interpretation_ids: Sequence[str],
) -> set[str]
Return the requested variants no committed record of product describes under the given IDs.
read_records
abstractmethod
¶
read_records(
model_id: str,
product: str,
*,
interpretation_ids: Sequence[str],
variant_ids: Sequence[str] | None = None,
) -> AsyncIterator[InterpretationRecord]
Stream committed records, optionally those describing any of variant_ids.
Records come in the order of interpretation_ids, then in commit order, then in put order.
locate
abstractmethod
async
¶
locate(record: InterpretationRecord) -> str
Return a location a consumer can read record's bytes from through the store's storage.
describe
abstractmethod
async
¶
describe(
model_id: str, interpretation_id: str
) -> StoredInterpretation | None
Return the stored descriptor and declared products of one interpretation ID, or None.
StoredInterpretation
dataclass
¶
StoredInterpretation(
model_id: str,
model_name: str,
descriptor: InterpretationDescriptor,
products: tuple[ArtifactSchema, ...],
)
A method run a store knows: its descriptor and the products declared under its interpretation ID.
StorageInterpretationStore
¶
StorageInterpretationStore(
storage: Storage,
root_uri: str,
index_path: str = ":memory:",
)
Bases: InterpretationStore
Product files in a Storage tree under root_uri; their index in SQLite at index_path.
index_path may be ":memory:" for tests, or the file a StorageArtifactStore uses.
locate
async
¶
locate(record: InterpretationRecord) -> str
Return the record's URI; the bytes are readable through this store's storage.
InterpretableModel
¶
Bases: ModelPlugin
A model binding that declares interpretation methods.
Subclass this instead of :class:~altar.models.ModelPlugin to opt in; core checks
isinstance(plugin, InterpretableModel) before preparing an interpretation. The binding declares its
methods, the scientific settings and runtimes each method's descriptor records, the plan that runs a chain,
and the reference span each row of a product file covers.
Provisional: the hooks are expected to change.
interpretation_methods
abstractmethod
¶
interpretation_methods() -> tuple[
InterpretationMethod, ...
]
Return every interpretation method the binding declares.
interpretation_scientific_dict
abstractmethod
¶
interpretation_scientific_dict(
configuration: ModelConfiguration,
*,
method: str,
target: str,
) -> Mapping[str, Any]
Return {"configuration": ..., "parameters": ...} for one method's descriptor.
configuration holds the model resources the method reads, by digest. parameters maps each
constant the plan passes on the method's scientific command lines to its text. Both exclude locations,
serialization formats and settings only another operation reads.
interpretation_runtime_identities
abstractmethod
¶
interpretation_runtime_identities(
configuration: ModelConfiguration, *, method: str
) -> Mapping[str, str]
Return the raw image of every runtime the method's tasks run, by the manifest's runtime role.
build_interpretation_workflow
abstractmethod
¶
build_interpretation_workflow(
request: InterpretationRequest,
*,
selection: InterpretationSelection,
variants_digest: str,
) -> InterpretationWorkflow
Build the plan that runs selection's method chain on the staged variant file.
The plan declares every product file as a terminal output, and declares the variant file with
variants_digest so a backend verifies it before a task reads it.
interpretation_rows
abstractmethod
¶
interpretation_rows(
*,
method: str,
product: str,
variants: Sequence[VariantKey],
) -> Sequence[InterpretationRow]
Return the rows of one product file written for variants, in the file's row order.
interpretation_input_span
¶
interpretation_input_span(
*, method: str, variant: VariantKey
) -> tuple[int, int] | None
Return the 0-based, half-open reference span method reads around variant, or None.
Preparation refuses a variant whose span runs off its contig, or whose contig the reference index lacks, before anything is computed. The default reads no span.
interpretation_variant_problem
¶
interpretation_variant_problem(
*, method: str, variant: VariantKey
) -> str | None
Return why method's runtime cannot interpret variant, or None when it can.
Preparation refuses every variant with a problem before anything is staged. Use it for a rule the runtime applies to each variant, such as the contigs it scores. The default refuses nothing.
interpretation_genome_index
¶
interpretation_genome_index(
run_inputs: ModelRunInputs,
) -> ResourceReference | None
Return the reference's .fai index from run_inputs, or None to read the one beside the FASTA.
InterpretationDescriptor
dataclass
¶
InterpretationDescriptor(
plugin_id: str,
architecture: str | None,
plugin_version: str,
method: str,
method_version: str,
target: str,
runtimes: Mapping[str, str],
configuration: Mapping[str, Any],
parameters: Mapping[str, str],
genome_build: str,
genome_hash: str | None = None,
upstream: str | None = None,
)
The unhashed inputs of one interpretation ID: everything that defines what a method run computes.
Core fills the plugin fields, the method and its version, the target, the upstream method's interpretation ID
and the genome. The binding supplies runtimes (runtime role to raw image), configuration (the model
resources the method reads, by digest) and parameters (every constant the binding passes on a scientific
command line). Operational settings, such as paths, run IDs and resource locations, are never part of it.
genome_hash hashes the genome's scientific identity, as a score lineage does.
:attr:interpretation_id hashes :meth:to_dict.
interpretation_id
property
¶
Return the frozen sha256: ID of this method run.
to_dict
¶
Return the descriptor as JSON values, as it is hashed and stored.
from_dict
classmethod
¶
from_dict(
data: Mapping[str, Any],
) -> InterpretationDescriptor
Rebuild a descriptor, rejecting unknown fields and other formats.
InterpretationEntry
dataclass
¶
InterpretationEntry(record: InterpretationRecord, row: int)
One committed product file that describes a variant, and the index of the variant's row in it.
interval
property
¶
interval: InterpretationRow
Return the variant's row: its reference span and anchor.
InterpretationIdentityError
¶
Bases: ValueError
A descriptor, operation or chain does not hash to the ID it states, or is not a valid chain.
InterpretationIngestResult
dataclass
¶
InterpretationIngestResult(
operation_id: str,
run_id: str,
committed: tuple[str, ...] = (),
skipped: Mapping[str, str] = dict(),
records: tuple[InterpretationRecord, ...] = (),
)
What one ingest stored: the methods it committed, the methods it skipped and why, and the records.
InterpretationMethod
dataclass
¶
InterpretationMethod(
name: str,
version: str,
description: str,
targets: tuple[str, ...],
products: tuple[ArtifactSchema, ...],
requires: str | None = None,
)
One interpretation method a binding declares: its version, the targets it explains and the files it writes.
targets is the closed set of model outputs the method can explain, such as "counts"; a request for any
other target is refused before compute. products are the files the method writes, declared as
:class:~altar.models.ArtifactSchema values. requires names at most one upstream method whose output this
method reads; requesting a method runs its whole chain.
InterpretationOperation
dataclass
¶
InterpretationOperation(
model_id: str,
descriptors: tuple[InterpretationDescriptor, ...],
variant_ids: tuple[str, ...],
plan_identity: PluginRunIdentity | None = None,
)
A durable record of one interpretation request: the method chain's descriptors and the input variants.
descriptors are in chain order, upstream first; the last one is the requested method. variant_ids
are canonical, unique and sorted. operation_id hashes the interpretation IDs and the inputs digest, so an
exact replay of a request computes the same ID. model_id partitions storage. plan_identity is the
exact identity the plan's tasks carry, kept as provenance. Neither is in the ID, and neither is the host's
run_id, which labels one execution.
interpretation_ids
property
¶
Return the chain's interpretation IDs, upstream first.
operation_id
property
¶
Return the sha256: ID of the chain's interpretation IDs and the input variants.
descriptor
¶
descriptor(method: str) -> InterpretationDescriptor
Return the descriptor of one method of the chain.
to_dict
¶
Serialize the operation, with every ID and digest stated beside what it hashes.
from_dict
classmethod
¶
from_dict(
data: Mapping[str, Any],
) -> InterpretationOperation
Rebuild an operation and check every stated ID and digest against what it hashes.
InterpretationPreparation
dataclass
¶
InterpretationPreparation(
operation: InterpretationOperation,
run_id: str,
variant_ids: tuple[str, ...],
workflow: InterpretationWorkflow | None,
variants_input: Transfer | None,
cached_count: int,
)
One prepared interpretation request.
variant_ids are the requested variants with a missing product, sorted; the plan runs exactly these.
workflow and variants_input are None when every variant is stored, and then nothing runs.
variants_input is the staged variant file at the location destination_transfer gave it, with its
digest. cached_count counts the requested variants whose every product is committed.
plan
property
¶
plan: ModelWorkflowPlan | None
Return the plan to execute, or None when nothing is missing.
InterpretationPreparationError
¶
Bases: ValueError
A request is refused before anything is computed: an unknown method or target, or an unusable variant.
InterpretationProductFile
dataclass
¶
InterpretationProductFile(
method: str,
product: str,
logical_path: str,
shard: tuple[tuple[str, str], ...] = (),
variant_id: str | None = None,
)
One product file a workflow plan writes: which method and product it is, and where the plan writes it.
logical_path is one of the plan's terminal outputs. shard holds labels that tell files of one product
apart, such as (("allele", "ref"),). variant_id is set only for a variant-scoped product.
InterpretationReadResult
dataclass
¶
InterpretationReadResult(
operation: InterpretationOperation,
variants: tuple[VariantInterpretation, ...],
missing: tuple[str, ...],
)
The stored interpretation of an operation's variants, and the variants with no complete execution.
InterpretationSelection
dataclass
¶
What a request asks for: one method, which runs its whole chain, and the target it explains.
InterpretationWorkflow
dataclass
¶
InterpretationWorkflow(
plan: ModelWorkflowPlan,
variants_file: str,
product_files: tuple[InterpretationProductFile, ...],
)
The plan a binding builds for one interpretation request, and how to read its outputs.
variants_file is the logical path of the staged variant file; every task that reads it declares it with
the digest the request gave. product_files names every file of every product of the chain's methods.
VariantInterpretation
dataclass
¶
VariantInterpretation(
variant_id: str,
run_id: str,
entries: tuple[InterpretationEntry, ...],
)
Every product of one variant, all from one execution, so motif hits pair with the contributions they used.
product
¶
product(name: str) -> tuple[InterpretationEntry, ...]
Return the entries of one product, such as the ref and alt files of an allele-sharded product.
InterpretationRunError
¶
InterpretationRunError(
message: str, result: InterpretationRunResult
)
Bases: RuntimeError
An interpretation execution failed, or one of its methods did not commit; result holds what it stored.
The methods that committed before the failure stay committed and readable through result.interpretation.
InterpretationRunResult
dataclass
¶
InterpretationRunResult(
preparation: InterpretationPreparation,
ingest: InterpretationIngestResult | None,
interpretation: InterpretationReadResult,
failure: str | None = None,
resumed: bool = False,
)
What :func:interpret prepared, ran, stored and read for one request.
ingest is None when every requested variant was already stored, so nothing ran. failure is the
workflow's error when a stage failed. resumed is true when the files an interrupted execution had
collected were ingested without running it again. interpretation is the operation read back from the
store: each variant's products, and the variants that still have none.
complete
property
¶
Return whether every requested variant has every product of the requested method chain.
require_interpretation_rows
¶
require_interpretation_rows(
schema: ArtifactSchema,
key: InterpretationKey,
rows: Sequence[InterpretationRow],
variant_ids: Sequence[str] | None = None,
) -> None
Reject rows that do not fit the product's scope, or that differ from the execution's variants.
A batch file's key names no variant and its rows describe distinct variants, one row each. A variant
file's key names one variant and the file has exactly one row, for that variant. With variant_ids, the
variants the execution ran, a batch file's rows must describe exactly those variants. Every store
applies the scope rule in :meth:InterpretationStore.put; ingest also passes variant_ids.
canonical_interpretation_variants
¶
canonical_interpretation_variants(
variant_ids: Iterable[str],
) -> tuple[VariantKey, ...]
Return the requested variants once each, sorted by canonical ID; refuse IDs that are not canonical.
ingest_interpretation_results
async
¶
ingest_interpretation_results(
plugin: ModelPlugin,
preparation: InterpretationPreparation,
*,
store: InterpretationStore,
storage: Storage,
output_transfer: TransferMapper | None = None,
) -> InterpretationIngestResult
Store and commit the product files an execution wrote, one method at a time in chain order.
output_transfer maps each plan output to where it was collected; pass the mapper given to
run_model_workflow. For each method, every declared file must exist, and its rows, computed by the
binding from the executed variants, must describe exactly those variants. Only then are the method's files
put and committed. A method with a missing or invalid file is skipped, with its reason, and so is every
method downstream of it; an upstream method that is complete still commits. Ingesting the same files again
stores nothing new and returns the same records.
interpretation_chain
¶
interpretation_chain(
plugin: ModelPlugin, method: str
) -> tuple[InterpretationMethod, ...]
Return the methods requesting method runs, upstream first.
The binding's methods must have unique names and products, and each requires must name another declared
method without a cycle. An unknown method raises :class:InterpretationPreparationError.
interpretation_descriptors
¶
interpretation_descriptors(
plugin: ModelPlugin,
configuration: ModelConfiguration
| Mapping[str, object],
run_inputs: ModelRunInputs | Mapping[str, object],
selection: InterpretationSelection,
) -> tuple[InterpretationDescriptor, ...]
Return the descriptors of selection's method chain, upstream first.
A target that a method of the chain does not declare raises :class:InterpretationPreparationError.
interpretation_inputs_digest
¶
Hash an operation's canonical, sorted input variant IDs.
prepare_interpretation
async
¶
prepare_interpretation(
plugin: ModelPlugin,
request: InterpretationRequest,
*,
selection: InterpretationSelection,
variant_ids: Iterable[str],
model_name: str,
store: InterpretationStore,
storage: Storage,
destination_transfer: TransferMapper,
) -> InterpretationPreparation
Record an interpretation request and build the plan for its variants with missing products.
request names the model, its configuration and run inputs, the execution's run_id and the backend's
path resolver. Every requested variant must be a canonical ID that the manifest admits, and that the binding's
InterpretableModel.interpretation_variant_problem accepts. For a run with a genome, its contig must be in the
reference index, the span the binding reads must lie inside the contig, and REF must equal the reference
bases. The index is the run's declared genome index or the .fai beside the FASTA, and the FASTA is read
through storage. Nothing is staged or written before these checks pass.
A variant is missing when any product of the chain has no committed record for it under its method's
interpretation ID. The binding's workflow is checked before the store is written: every product file must be a
plan output, and every image and digest the tasks use must be in a descriptor. The function then declares each
product and records the operation in store. With no misses the preparation has no plan. Otherwise the
missing variants, sorted, are written as the variant file at the location destination_transfer maps the
plan's variant path to, and the plan stages that file with its digest. Pass the same mapper as
run_model_workflow's input_transfer.
To resume an ingest that stopped before it committed, prepare again with the same run_id and ingest the
collected files again.
read_interpretation
async
¶
read_interpretation(
store: InterpretationStore,
operation: InterpretationOperation,
*,
variant_ids: Iterable[str] | None = None,
) -> InterpretationReadResult
Return each variant's committed products under the operation's interpretation IDs.
variant_ids defaults to the operation's inputs. For each variant, the earliest execution whose commits
cover every product of every method in the chain supplies all of them. A variant with no such execution is
listed in missing. The store's records are read, never their files.
verify_interpretation_descriptor
¶
verify_interpretation_descriptor(
data: Mapping[str, Any], interpretation_id: str
) -> InterpretationDescriptor
Rebuild a stored descriptor and check that it hashes to interpretation_id.
Raises :class:InterpretationIdentityError for a descriptor that was changed after its ID was recorded.
interpret
async
¶
interpret(
plugin: ModelPlugin,
request: InterpretationRequest,
*,
selection: InterpretationSelection,
variant_ids: Iterable[str],
model_name: str,
store: InterpretationStore,
storage: Storage,
backend: ExecutionBackend,
input_transfer: TransferMapper,
output_transfer: TransferMapper,
spec_mapper: SpecMapper | None = None,
checkpoint_uri: str | None = None,
poll_interval_s: float = 5.0,
timeout_s: float = 86400.0,
progress: Callable[[str], None] | None = None,
) -> InterpretationRunResult
Interpret variant_ids with selection's method chain, store every product, and read them back.
request names the model, its configuration and run inputs, the execution's run_id and the backend's
path resolver. input_transfer maps a file the platform stages, which a binding declares with its logical
path as URI, to a storage location; it places the variant file and is the workflow's input mapper, and it must
leave a model resource's URI alone. output_transfer maps each product file to where it is collected.
The steps are:
- :func:
~altar.interpretation.prepare_interpretationrefuses unusable variants, records the operation and builds the plan for the variants with a missing product. With none missing, nothing runs. - Every collected location is cleared, and the plan runs on
backend. When a stage fails, the files of the stages that finished are still collected. - :func:
~altar.interpretation.ingest_interpretation_resultscommits each method whose files are complete, upstream first. - :func:
~altar.interpretation.read_interpretationreads the operation back.
With checkpoint_uri, a JSON file there records the execution's progress (:class:InterpretationCheckpoint).
Calling again with the same run_id then resumes: an execution stopped while its tasks ran is run again, and
one stopped after collection ingests the files it collected, if their digests still match, without running.
The run_id of an execution that committed products cannot run again: preparation then raises
:class:~altar.interpretation.InterpretationPreparationError before anything is submitted. A backend that may
adopt a running task must also clear outputs on a fresh start, as run_scoring requires.
poll_interval_s, timeout_s and progress are :func:~altar.models.run_model_workflow's: the time
limit applies to each wave of stages, and a wave that reaches it fails the execution like a failed stage, with
the workflow's error as the cause of the :class:InterpretationRunError. Before the plan runs, progress
also receives the number of variants to interpret.
Raises :class:InterpretationRunError, whose result holds what was stored, when a stage fails or a method
does not commit.