Skip to content

Interpretation

altar.interpretation runs interpretation operations, such as ChromBPNet's DeepSHAP contributions, and stores their products. Bindings that declare interpretation methods import from it too. interpret() runs one request on an execution backend. An application that drives execution itself calls prepare_interpretation(), ingest_interpretation_results() and read_interpretation(), as Run and read interpretations shows.

Provisional: every name in this module is provisional. See Provisional names.

interpretation

Interpretation operations: explain a model's predictions for variants, and store what each method writes.

A binding opts in by subclassing :class:InterpretableModel and declaring :class:InterpretationMethod values. :func:interpret runs a request in one call on an execution backend. A host that drives execution itself prepares a request with :func:prepare_interpretation, runs the plan with :func:~altar.models.run_model_workflow, stores the product files with :func:ingest_interpretation_results, and reads them back with :func:read_interpretation. :class:StorageInterpretationStore is the reference :class:InterpretationStore.

Every name here is provisional and listed in __provisional__: it is expected to change. Stored interpretation records are keyed by their interpretation ID.

InterpretationCommitError

Bases: ValueError

A commit, or a put that a commit would rely on, breaks the store's rules; nothing was made visible.

InterpretationKey dataclass

InterpretationKey(
    run_id: str,
    shard: tuple[tuple[str, str], ...] = (),
    variant_id: str | None = None,
)

The execution that wrote one product file, and the variant a per-variant file is for.

run_id labels one execution of an interpretation plan; the host chooses it. shard holds further labels that tell a method's files of one product apart, such as ("allele", "ref"), as sorted pairs. variant_id is set only for a variant-scoped product.

digest property

digest: str

Return the canonical key hash a store may index.

InterpretationRecord dataclass

InterpretationRecord(
    model_id: str,
    product: str,
    interpretation_id: str,
    key: InterpretationKey,
    rows: tuple[InterpretationRow, ...],
    media_type: str,
    digest: str,
    byte_size: int,
    uri: str,
    operation_id: str | None = None,
)

One stored product file: the method run and execution that wrote it, its rows, and where its bytes are.

operation_id is the operation whose commit made the record visible. A record :meth:InterpretationStore.put returns is not committed yet, so its operation_id is None. rows follow the rows of the file in order. digest, byte_size and uri are as on :class:~altar.results.ArtifactRecord.

variant_ids property

variant_ids: tuple[str, ...]

Return the variants the file's rows describe, in row order.

row_of

row_of(variant_id: str) -> int

Return the index of variant_id's row in the file; raise KeyError when no row describes it.

InterpretationRow dataclass

InterpretationRow(
    variant_id: str,
    chrom: str,
    start: int,
    end: int,
    anchor: int,
)

The variant one row of a product file describes, and the reference span the row's data covers.

start and end are a 0-based, half-open interval on chrom. anchor is the offset of the variant's first base in that interval, so start + anchor is the variant's 0-based position. A row of a per-base array maps its element i to reference base start + i; the binding documents how bases after an indel's alternate allele map.

InterpretationStore

Bases: ABC

Persistence axis for interpretation products.

A store keeps the operations it was given, the descriptor and product schemas of each interpretation ID, the files put under them, and the commits that made files visible. Reads take the interpretation IDs whose records count, in preference order. The store never decodes a file.

Provisional: the contract is expected to change.

close

close() -> None

Release the connections and clients this store opened itself; a second call does nothing.

record_operation abstractmethod async

record_operation(
    operation: InterpretationOperation,
) -> None

Record one operation, idempotently by its operation_id.

The store checks that every descriptor hashes to its interpretation ID and that the operation ID matches its contents, and raises :class:~altar.interpretation.InterpretationIdentityError otherwise. A second operation with the same ID keeps the first record.

read_operation abstractmethod async

read_operation(
    model_id: str, operation_id: str
) -> InterpretationOperation | None

Return the recorded operation, verified again, or None when the store has none.

ensure_product abstractmethod async

ensure_product(
    *,
    model_id: str,
    model_name: str,
    descriptor: InterpretationDescriptor,
    schema: ArtifactSchema,
) -> None

Declare one product under the descriptor's interpretation ID.

The descriptor is stored unhashed. A product already declared under the ID with another name, media type or scope raises :class:~altar.results.SchemaCollisionError.

put abstractmethod async

put(
    model_id: str,
    product: str,
    *,
    interpretation_id: str,
    key: InterpretationKey,
    rows: Sequence[InterpretationRow],
    source_uri: str,
    storage: Storage,
    digest: str,
) -> InterpretationRecord

Copy the file at source_uri (readable through storage) into the store and index it, uncommitted.

The product must have been declared with :meth:ensure_product first, and key and rows must fit its scope (:func:require_interpretation_rows). digest is the sha256: identity of the bytes, which the store checks. A put of bytes the store already holds under the same key returns the existing record. Different bytes under the same key are another record.

commit abstractmethod async

commit(
    model_id: str,
    *,
    operation_id: str,
    run_id: str,
    method: str,
    records: Sequence[InterpretationRecord],
) -> tuple[InterpretationRecord, ...]

Make one method's records of one execution visible together, and return them as committed.

The operation must be recorded and list method. Every record must have been put under the method's interpretation ID and keyed by run_id, with one record per product and key. Every product declared under the ID must have records, and each product's rows must describe the same variants. A method with an upstream method commits only after that method committed in the same execution, over the same variants. An execution belongs to one operation. Committing the same records again returns them; committing other records for the same operation, execution and method raises :class:InterpretationCommitError.

get_missing_variants abstractmethod async

get_missing_variants(
    model_id: str,
    product: str,
    variant_ids: Sequence[str],
    interpretation_ids: Sequence[str],
) -> set[str]

Return the requested variants no committed record of product describes under the given IDs.

read_records abstractmethod

read_records(
    model_id: str,
    product: str,
    *,
    interpretation_ids: Sequence[str],
    variant_ids: Sequence[str] | None = None,
) -> AsyncIterator[InterpretationRecord]

Stream committed records, optionally those describing any of variant_ids.

Records come in the order of interpretation_ids, then in commit order, then in put order.

locate abstractmethod async

locate(record: InterpretationRecord) -> str

Return a location a consumer can read record's bytes from through the store's storage.

describe abstractmethod async

describe(
    model_id: str, interpretation_id: str
) -> StoredInterpretation | None

Return the stored descriptor and declared products of one interpretation ID, or None.

StoredInterpretation dataclass

StoredInterpretation(
    model_id: str,
    model_name: str,
    descriptor: InterpretationDescriptor,
    products: tuple[ArtifactSchema, ...],
)

A method run a store knows: its descriptor and the products declared under its interpretation ID.

StorageInterpretationStore

StorageInterpretationStore(
    storage: Storage,
    root_uri: str,
    index_path: str = ":memory:",
)

Bases: InterpretationStore

Product files in a Storage tree under root_uri; their index in SQLite at index_path.

index_path may be ":memory:" for tests, or the file a StorageArtifactStore uses.

locate async

locate(record: InterpretationRecord) -> str

Return the record's URI; the bytes are readable through this store's storage.

close

close() -> None

Close the index's SQLite connection; a second call does nothing.

InterpretableModel

InterpretableModel()

Bases: ModelPlugin

A model binding that declares interpretation methods.

Subclass this instead of :class:~altar.models.ModelPlugin to opt in; core checks isinstance(plugin, InterpretableModel) before preparing an interpretation. The binding declares its methods, the scientific settings and runtimes each method's descriptor records, the plan that runs a chain, and the reference span each row of a product file covers.

Provisional: the hooks are expected to change.

interpretation_methods abstractmethod

interpretation_methods() -> tuple[
    InterpretationMethod, ...
]

Return every interpretation method the binding declares.

interpretation_scientific_dict abstractmethod

interpretation_scientific_dict(
    configuration: ModelConfiguration,
    *,
    method: str,
    target: str,
) -> Mapping[str, Any]

Return {"configuration": ..., "parameters": ...} for one method's descriptor.

configuration holds the model resources the method reads, by digest. parameters maps each constant the plan passes on the method's scientific command lines to its text. Both exclude locations, serialization formats and settings only another operation reads.

interpretation_runtime_identities abstractmethod

interpretation_runtime_identities(
    configuration: ModelConfiguration, *, method: str
) -> Mapping[str, str]

Return the raw image of every runtime the method's tasks run, by the manifest's runtime role.

build_interpretation_workflow abstractmethod

build_interpretation_workflow(
    request: InterpretationRequest,
    *,
    selection: InterpretationSelection,
    variants_digest: str,
) -> InterpretationWorkflow

Build the plan that runs selection's method chain on the staged variant file.

The plan declares every product file as a terminal output, and declares the variant file with variants_digest so a backend verifies it before a task reads it.

interpretation_rows abstractmethod

interpretation_rows(
    *,
    method: str,
    product: str,
    variants: Sequence[VariantKey],
) -> Sequence[InterpretationRow]

Return the rows of one product file written for variants, in the file's row order.

interpretation_input_span

interpretation_input_span(
    *, method: str, variant: VariantKey
) -> tuple[int, int] | None

Return the 0-based, half-open reference span method reads around variant, or None.

Preparation refuses a variant whose span runs off its contig, or whose contig the reference index lacks, before anything is computed. The default reads no span.

interpretation_variant_problem

interpretation_variant_problem(
    *, method: str, variant: VariantKey
) -> str | None

Return why method's runtime cannot interpret variant, or None when it can.

Preparation refuses every variant with a problem before anything is staged. Use it for a rule the runtime applies to each variant, such as the contigs it scores. The default refuses nothing.

interpretation_genome_index

interpretation_genome_index(
    run_inputs: ModelRunInputs,
) -> ResourceReference | None

Return the reference's .fai index from run_inputs, or None to read the one beside the FASTA.

InterpretationDescriptor dataclass

InterpretationDescriptor(
    plugin_id: str,
    architecture: str | None,
    plugin_version: str,
    method: str,
    method_version: str,
    target: str,
    runtimes: Mapping[str, str],
    configuration: Mapping[str, Any],
    parameters: Mapping[str, str],
    genome_build: str,
    genome_hash: str | None = None,
    upstream: str | None = None,
)

The unhashed inputs of one interpretation ID: everything that defines what a method run computes.

Core fills the plugin fields, the method and its version, the target, the upstream method's interpretation ID and the genome. The binding supplies runtimes (runtime role to raw image), configuration (the model resources the method reads, by digest) and parameters (every constant the binding passes on a scientific command line). Operational settings, such as paths, run IDs and resource locations, are never part of it. genome_hash hashes the genome's scientific identity, as a score lineage does.

:attr:interpretation_id hashes :meth:to_dict.

interpretation_id property

interpretation_id: str

Return the frozen sha256: ID of this method run.

to_dict

to_dict() -> dict[str, Any]

Return the descriptor as JSON values, as it is hashed and stored.

from_dict classmethod

from_dict(
    data: Mapping[str, Any],
) -> InterpretationDescriptor

Rebuild a descriptor, rejecting unknown fields and other formats.

InterpretationEntry dataclass

InterpretationEntry(record: InterpretationRecord, row: int)

One committed product file that describes a variant, and the index of the variant's row in it.

interval property

Return the variant's row: its reference span and anchor.

InterpretationIdentityError

Bases: ValueError

A descriptor, operation or chain does not hash to the ID it states, or is not a valid chain.

InterpretationIngestResult dataclass

InterpretationIngestResult(
    operation_id: str,
    run_id: str,
    committed: tuple[str, ...] = (),
    skipped: Mapping[str, str] = dict(),
    records: tuple[InterpretationRecord, ...] = (),
)

What one ingest stored: the methods it committed, the methods it skipped and why, and the records.

InterpretationMethod dataclass

InterpretationMethod(
    name: str,
    version: str,
    description: str,
    targets: tuple[str, ...],
    products: tuple[ArtifactSchema, ...],
    requires: str | None = None,
)

One interpretation method a binding declares: its version, the targets it explains and the files it writes.

targets is the closed set of model outputs the method can explain, such as "counts"; a request for any other target is refused before compute. products are the files the method writes, declared as :class:~altar.models.ArtifactSchema values. requires names at most one upstream method whose output this method reads; requesting a method runs its whole chain.

product

product(name: str) -> ArtifactSchema

Return the declared product called name.

InterpretationOperation dataclass

InterpretationOperation(
    model_id: str,
    descriptors: tuple[InterpretationDescriptor, ...],
    variant_ids: tuple[str, ...],
    plan_identity: PluginRunIdentity | None = None,
)

A durable record of one interpretation request: the method chain's descriptors and the input variants.

descriptors are in chain order, upstream first; the last one is the requested method. variant_ids are canonical, unique and sorted. operation_id hashes the interpretation IDs and the inputs digest, so an exact replay of a request computes the same ID. model_id partitions storage. plan_identity is the exact identity the plan's tasks carry, kept as provenance. Neither is in the ID, and neither is the host's run_id, which labels one execution.

method property

method: str

Return the requested method: the last of the chain.

methods property

methods: tuple[str, ...]

Return the chain's method names, upstream first.

interpretation_ids property

interpretation_ids: tuple[str, ...]

Return the chain's interpretation IDs, upstream first.

operation_id property

operation_id: str

Return the sha256: ID of the chain's interpretation IDs and the input variants.

descriptor

descriptor(method: str) -> InterpretationDescriptor

Return the descriptor of one method of the chain.

to_dict

to_dict() -> dict[str, Any]

Serialize the operation, with every ID and digest stated beside what it hashes.

from_dict classmethod

from_dict(
    data: Mapping[str, Any],
) -> InterpretationOperation

Rebuild an operation and check every stated ID and digest against what it hashes.

InterpretationPreparation dataclass

InterpretationPreparation(
    operation: InterpretationOperation,
    run_id: str,
    variant_ids: tuple[str, ...],
    workflow: InterpretationWorkflow | None,
    variants_input: Transfer | None,
    cached_count: int,
)

One prepared interpretation request.

variant_ids are the requested variants with a missing product, sorted; the plan runs exactly these. workflow and variants_input are None when every variant is stored, and then nothing runs. variants_input is the staged variant file at the location destination_transfer gave it, with its digest. cached_count counts the requested variants whose every product is committed.

plan property

plan: ModelWorkflowPlan | None

Return the plan to execute, or None when nothing is missing.

InterpretationPreparationError

Bases: ValueError

A request is refused before anything is computed: an unknown method or target, or an unusable variant.

InterpretationProductFile dataclass

InterpretationProductFile(
    method: str,
    product: str,
    logical_path: str,
    shard: tuple[tuple[str, str], ...] = (),
    variant_id: str | None = None,
)

One product file a workflow plan writes: which method and product it is, and where the plan writes it.

logical_path is one of the plan's terminal outputs. shard holds labels that tell files of one product apart, such as (("allele", "ref"),). variant_id is set only for a variant-scoped product.

key

key(run_id: str) -> InterpretationKey

Return the store key of this file in execution run_id.

InterpretationReadResult dataclass

InterpretationReadResult(
    operation: InterpretationOperation,
    variants: tuple[VariantInterpretation, ...],
    missing: tuple[str, ...],
)

The stored interpretation of an operation's variants, and the variants with no complete execution.

InterpretationSelection dataclass

InterpretationSelection(method: str, target: str)

What a request asks for: one method, which runs its whole chain, and the target it explains.

InterpretationWorkflow dataclass

InterpretationWorkflow(
    plan: ModelWorkflowPlan,
    variants_file: str,
    product_files: tuple[InterpretationProductFile, ...],
)

The plan a binding builds for one interpretation request, and how to read its outputs.

variants_file is the logical path of the staged variant file; every task that reads it declares it with the digest the request gave. product_files names every file of every product of the chain's methods.

VariantInterpretation dataclass

VariantInterpretation(
    variant_id: str,
    run_id: str,
    entries: tuple[InterpretationEntry, ...],
)

Every product of one variant, all from one execution, so motif hits pair with the contributions they used.

product

product(name: str) -> tuple[InterpretationEntry, ...]

Return the entries of one product, such as the ref and alt files of an allele-sharded product.

InterpretationRunError

InterpretationRunError(
    message: str, result: InterpretationRunResult
)

Bases: RuntimeError

An interpretation execution failed, or one of its methods did not commit; result holds what it stored.

The methods that committed before the failure stay committed and readable through result.interpretation.

InterpretationRunResult dataclass

InterpretationRunResult(
    preparation: InterpretationPreparation,
    ingest: InterpretationIngestResult | None,
    interpretation: InterpretationReadResult,
    failure: str | None = None,
    resumed: bool = False,
)

What :func:interpret prepared, ran, stored and read for one request.

ingest is None when every requested variant was already stored, so nothing ran. failure is the workflow's error when a stage failed. resumed is true when the files an interrupted execution had collected were ingested without running it again. interpretation is the operation read back from the store: each variant's products, and the variants that still have none.

complete property

complete: bool

Return whether every requested variant has every product of the requested method chain.

require_interpretation_rows

require_interpretation_rows(
    schema: ArtifactSchema,
    key: InterpretationKey,
    rows: Sequence[InterpretationRow],
    variant_ids: Sequence[str] | None = None,
) -> None

Reject rows that do not fit the product's scope, or that differ from the execution's variants.

A batch file's key names no variant and its rows describe distinct variants, one row each. A variant file's key names one variant and the file has exactly one row, for that variant. With variant_ids, the variants the execution ran, a batch file's rows must describe exactly those variants. Every store applies the scope rule in :meth:InterpretationStore.put; ingest also passes variant_ids.

canonical_interpretation_variants

canonical_interpretation_variants(
    variant_ids: Iterable[str],
) -> tuple[VariantKey, ...]

Return the requested variants once each, sorted by canonical ID; refuse IDs that are not canonical.

ingest_interpretation_results async

ingest_interpretation_results(
    plugin: ModelPlugin,
    preparation: InterpretationPreparation,
    *,
    store: InterpretationStore,
    storage: Storage,
    output_transfer: TransferMapper | None = None,
) -> InterpretationIngestResult

Store and commit the product files an execution wrote, one method at a time in chain order.

output_transfer maps each plan output to where it was collected; pass the mapper given to run_model_workflow. For each method, every declared file must exist, and its rows, computed by the binding from the executed variants, must describe exactly those variants. Only then are the method's files put and committed. A method with a missing or invalid file is skipped, with its reason, and so is every method downstream of it; an upstream method that is complete still commits. Ingesting the same files again stores nothing new and returns the same records.

interpretation_chain

interpretation_chain(
    plugin: ModelPlugin, method: str
) -> tuple[InterpretationMethod, ...]

Return the methods requesting method runs, upstream first.

The binding's methods must have unique names and products, and each requires must name another declared method without a cycle. An unknown method raises :class:InterpretationPreparationError.

interpretation_descriptors

interpretation_descriptors(
    plugin: ModelPlugin,
    configuration: ModelConfiguration
    | Mapping[str, object],
    run_inputs: ModelRunInputs | Mapping[str, object],
    selection: InterpretationSelection,
) -> tuple[InterpretationDescriptor, ...]

Return the descriptors of selection's method chain, upstream first.

A target that a method of the chain does not declare raises :class:InterpretationPreparationError.

interpretation_inputs_digest

interpretation_inputs_digest(
    variant_ids: Sequence[str],
) -> str

Hash an operation's canonical, sorted input variant IDs.

prepare_interpretation async

prepare_interpretation(
    plugin: ModelPlugin,
    request: InterpretationRequest,
    *,
    selection: InterpretationSelection,
    variant_ids: Iterable[str],
    model_name: str,
    store: InterpretationStore,
    storage: Storage,
    destination_transfer: TransferMapper,
) -> InterpretationPreparation

Record an interpretation request and build the plan for its variants with missing products.

request names the model, its configuration and run inputs, the execution's run_id and the backend's path resolver. Every requested variant must be a canonical ID that the manifest admits, and that the binding's InterpretableModel.interpretation_variant_problem accepts. For a run with a genome, its contig must be in the reference index, the span the binding reads must lie inside the contig, and REF must equal the reference bases. The index is the run's declared genome index or the .fai beside the FASTA, and the FASTA is read through storage. Nothing is staged or written before these checks pass.

A variant is missing when any product of the chain has no committed record for it under its method's interpretation ID. The binding's workflow is checked before the store is written: every product file must be a plan output, and every image and digest the tasks use must be in a descriptor. The function then declares each product and records the operation in store. With no misses the preparation has no plan. Otherwise the missing variants, sorted, are written as the variant file at the location destination_transfer maps the plan's variant path to, and the plan stages that file with its digest. Pass the same mapper as run_model_workflow's input_transfer.

To resume an ingest that stopped before it committed, prepare again with the same run_id and ingest the collected files again.

read_interpretation async

read_interpretation(
    store: InterpretationStore,
    operation: InterpretationOperation,
    *,
    variant_ids: Iterable[str] | None = None,
) -> InterpretationReadResult

Return each variant's committed products under the operation's interpretation IDs.

variant_ids defaults to the operation's inputs. For each variant, the earliest execution whose commits cover every product of every method in the chain supplies all of them. A variant with no such execution is listed in missing. The store's records are read, never their files.

verify_interpretation_descriptor

verify_interpretation_descriptor(
    data: Mapping[str, Any], interpretation_id: str
) -> InterpretationDescriptor

Rebuild a stored descriptor and check that it hashes to interpretation_id.

Raises :class:InterpretationIdentityError for a descriptor that was changed after its ID was recorded.

interpret async

interpret(
    plugin: ModelPlugin,
    request: InterpretationRequest,
    *,
    selection: InterpretationSelection,
    variant_ids: Iterable[str],
    model_name: str,
    store: InterpretationStore,
    storage: Storage,
    backend: ExecutionBackend,
    input_transfer: TransferMapper,
    output_transfer: TransferMapper,
    spec_mapper: SpecMapper | None = None,
    checkpoint_uri: str | None = None,
    poll_interval_s: float = 5.0,
    timeout_s: float = 86400.0,
    progress: Callable[[str], None] | None = None,
) -> InterpretationRunResult

Interpret variant_ids with selection's method chain, store every product, and read them back.

request names the model, its configuration and run inputs, the execution's run_id and the backend's path resolver. input_transfer maps a file the platform stages, which a binding declares with its logical path as URI, to a storage location; it places the variant file and is the workflow's input mapper, and it must leave a model resource's URI alone. output_transfer maps each product file to where it is collected.

The steps are:

  1. :func:~altar.interpretation.prepare_interpretation refuses unusable variants, records the operation and builds the plan for the variants with a missing product. With none missing, nothing runs.
  2. Every collected location is cleared, and the plan runs on backend. When a stage fails, the files of the stages that finished are still collected.
  3. :func:~altar.interpretation.ingest_interpretation_results commits each method whose files are complete, upstream first.
  4. :func:~altar.interpretation.read_interpretation reads the operation back.

With checkpoint_uri, a JSON file there records the execution's progress (:class:InterpretationCheckpoint). Calling again with the same run_id then resumes: an execution stopped while its tasks ran is run again, and one stopped after collection ingests the files it collected, if their digests still match, without running. The run_id of an execution that committed products cannot run again: preparation then raises :class:~altar.interpretation.InterpretationPreparationError before anything is submitted. A backend that may adopt a running task must also clear outputs on a fresh start, as run_scoring requires.

poll_interval_s, timeout_s and progress are :func:~altar.models.run_model_workflow's: the time limit applies to each wave of stages, and a wave that reaches it fails the execution like a failed stage, with the workflow's error as the cause of the :class:InterpretationRunError. Before the plan runs, progress also receives the number of variants to interpret.

Raises :class:InterpretationRunError, whose result holds what was stored, when a stage fails or a method does not commit.