Skip to content

Upgrade from 0.3 to 0.4

Most changes from Altar 0.3 to 0.4 need no action. The ones that do affect run specs that set [backend] max_ticks, work_dir or reference_digest, scripts that read the commands' standard error, stores that hold AlphaGenome or LegNet scores, ChromBPNet interpretation results that your own application caches, scripts that call the dataset builders, code that calls the scoring drivers with max_ticks, and custom execution backends.

Before you upgrade

  • Upgrade altar, altar-identity and every binding together. Each 0.4 binding requires altar>=0.4.0,<0.5.
  • Altar needs Python 3.12 or newer; 3.12 and 3.13 are tested. See Platform support.
  • Score lineages are the same in 0.3 and 0.4, with one exception: an AlphaGenome run given a reference genome now has the lineage of the same run without one. AlphaGenome and the reference genome explains what to do. Every other stored score stays a cache hit after the upgrade.

Changes that need action

Run specs

Change Who is affected What to do
[backend] max_ticks, a count of checks, is replaced by timeout, a time limit for each batch of a model or each wave of workflow stages. It takes seconds or text such as "90m", "24h" or "2d", and defaults to 24 hours. A spec that sets max_ticks is refused Specs that set max_ticks Replace it with timeout. See The tasks ran too long for what happens to open tasks at the limit on each backend
[backend] poll_interval_s defaults to 5 seconds instead of 1 Specs that rely on the default Nothing, unless you want quicker checks: set poll_interval_s
A run keeps its files in two directories. [run] work_dir, now optional with a default of ./<job_id>, holds only this run's files and the export. A new [run] cache_dir, $ALTAR_CACHE_DIR or ./.altar by default, holds what later runs reuse: prepared files under staging/, the default artifact store under artifacts/, interpretations and fetched entries. In 0.3, prepared files and the default artifact store were under work_dir Specs whose models read prepared files, such as ChromBPNet, or keep artifacts in the default store Set cache_dir to the old work_dir to keep using what is there, or run altar prepare again. See the run section
Without work_dir, a run refuses to start when ./<job_id> already holds files another run did not write Specs that leave out work_dir Set work_dir, or choose another job_id. See The run directory already exists
The local backend's default data_root is backend-data in the cache. ALTAR_LOCAL_DATA_ROOT is no longer read, and altar.plugins.execution.DATA_ROOT_ENV is removed. Without a data_root, LocalDockerExecutionBackend uses $ALTAR_CACHE_DIR/backend-data, or ./.altar/backend-data, in place of ./.altar-local-data Anyone who set ALTAR_LOCAL_DATA_ROOT, or reads files under .altar-local-data Set [backend] options = { data_root = "..." }, or pass data_root= in Python
A task that asks for a GPU pool its backend does not define is refused before anything is submitted, with UnknownPoolError. The local backend defines gpu, or the names in its new pools option; kubernetes defines gpu and the names in pool_node_selectors and max_slots; modal defines the keys of pool_accelerators Specs and bindings that ask for a pool name a backend does not list Add the pool to the backend's options, or ask for one it defines. See GPU pools
The modal backend no longer upper-cases a pool name it does not map into an accelerator name Modal users whose tasks ask for a pool missing from pool_accelerators, such as a100 Map the pool in pool_accelerators, such as { gpu = "A10G", a100 = "A100" }. Its keys replace the default, so keep gpu if your tasks use it
[variants] reference_digest is checked against the FASTA on every run, before any lookup. A mismatch stops the run Anyone whose spec sets reference_digest Make sure the digest is the FASTA's. Change it only if you meant to change the reference: models that read the reference then score under a new lineage
The first run with reference_digest writes <fasta>.sha256-verified beside the FASTA Anyone whose spec sets reference_digest Nothing. Later runs read this record instead of hashing an unchanged file. If the run cannot write in the FASTA's directory, it hashes the file on every run
A spec that sets reference_digest without reference_fasta is refused Specs that set only the digest Add reference_fasta, or remove the digest
A run whose model needs an annotation source for its prioritization rule now stops before scoring if the spec lacks that source. It used to fail at export Specs that score such a model without its source Add the [sources.<name>] section the error names

Command output

Change Who is affected What to do
Every altar command prints standard error in one format. A progress line has no altar run: prefix and starts with its spec section, such as [models.<name>]. A problem starts with altar <command>: warning: or altar <command>: error:, and continues on lines indented two spaces. Each variant a lookup may miss is its own warning. Counts agree in number instead of (s), and altar pull prints sizes in decimal units. Standard output and its JSON are unchanged Scripts that read the commands' standard error Match the new lines, or read the JSON on standard output. See the command-line reference

Stored results

Change Who is affected What to do
An AlphaGenome run given a reference has the lineage of the run without one Stores that hold AlphaGenome scores from runs with reference_fasta See AlphaGenome and the reference genome
LegNet lineages that differ only in cell_type or assay reuse each other's scores Anyone who kept such lineages apart on purpose Nothing; their scores are identical. Read with ScoreReusePolicy.exactly(...) to keep a fixed set
LegNetConfiguration.from_resource_manifest refuses a cell_type that differs from the manifest's, and the official LegNet weights must be labelled WTC11 and LentiMPRA Callers that relabelled a checkpoint Leave cell_type out, or correct the manifest. See LegNet labels
ChromBPNet's interpretation runs the seeded DeepSHAP method, deepshap_counts 2.0.0, on its own pinned image, and its motif stage runs riyavsinha/altar-finemo, which runs as a non-root user. build_interpretation_plan uses them too, so its interpretation run IDs change Applications that cache ChromBPNet interpretation results by run ID Start a new interpretation run after the upgrade. Results stored under the old IDs are not reused, and the two methods are not treated as the same. Score lineages do not change

Python code and extensions

Change Who is affected What to do
run_scoring and score_cohort take timeout_s (default 86,400 seconds) in place of max_ticks, and run_model_workflow takes it in place of max_ticks_per_wave. Their poll_interval_s defaults to 5 seconds instead of 1. Tasks still open at the limit are cancelled and TimeoutError is raised Code that passes max_ticks or max_ticks_per_wave Pass timeout_s, in seconds. run_reconcile_loop keeps max_ticks and also takes timeout_s
ReconcileResult has two new fields, running and pending, after failed Code that builds a ReconcileResult with positional arguments Pass its fields by name
RunPlan has a new cache_dir field after work_dir Code that builds a RunPlan with positional arguments Pass its fields by name. RunPlan is Provisional
The kubernetes backend reports an active Job whose pod is not ready yet as PENDING, not RUNNING. A cluster that does not report status.ready, before Kubernetes 1.24, still reports every active pod as RUNNING Code that reads task states from KubernetesExecutionBackend Treat PENDING as a task that has not started: it may be scheduling or pulling its image
Deprecated builder flags: altar-popfreq-build --version, altar-clinvar-build --release-date, and --source-release on altar-alphamissense-build and altar-spliceai-build. altar-e2g-atlas-ingest takes --input and --database in place of two positional arguments Scripts that build source datasets Use --release, and --input … --database …. The old forms still work in 0.4 and print a warning
run_scoring refuses a backend that adopts running tasks (adopts_active_tasks) without declaring clears_outputs_on_fresh_start. A subclass that overrides submit resets that declaration to False Custom execution backends that adopt tasks, and subclasses of KubernetesExecutionBackend that override submit Clear each output before a fresh attempt starts, and declare clears_outputs_on_fresh_start = True. See Clear outputs an earlier run left
New conformance tests: the execution-backend suite checks output clearing on a fresh start, an adopted task and a replaced task, and the refusal of an undefined GPU pool; the storage suite checks failed and interrupted writes; the store and source suites check that close() can be called twice Authors of backends, storage adapters, stores and sources Run the suites. A backend that declares adopts_active_tasks or clears_outputs_on_fresh_start implements the suite's write_task_output and task_output hooks. A backend that defines its GPU pools implements gpu_pools(); see Declare your GPU pools
LocalStorage and FsspecStorage write a local file through a temporary file and a rename. A destination that is a symbolic link is replaced, not written through Code that writes to a symbolic link through Altar's storage Write to the link's target path instead

AlphaGenome and the reference genome

AlphaGenome never reads a local reference: the hosted API supplies its own sequence. In 0.3, a run given [variants] reference_fasta still recorded the FASTA's digest in AlphaGenome's lineage, so adding a FASTA for left-alignment made every stored AlphaGenome score a cache miss. In 0.4, AlphaGenome's run inputs carry no reference, and its lineage records the genome build only. A run that carried a reference therefore moves to the lineage of the same run without one. For the binding's human_defaults fixture:

Run Lineage ID in 0.3 Lineage ID in 0.4
No reference d92e7b1c… d92e7b1c…
Reference with digest sha256:aaaa…, at any path ad43d218… d92e7b1c…
Reference with digest sha256:bbbb… 657bc3b8… d92e7b1c…
Reference without a digest 3bf3b854… d92e7b1c…

AlphaGenome's run identity no longer carries genome_hash. AlphaGenome's equivalence rules drop the reference from an older lineage's descriptor, so scores stored under a lineage that recorded a reference stay reachable in a store that keeps lineages. The rules ID is now fbf5b668… (rules format 2).

  • Lineages written by 0.3 have descriptors. Nothing to do: the next resolution lists the older lineage as equivalent, and its rows are hits.
  • Lineages written before 0.3 have no descriptor. Attach one, rebuilt as AlphaGenome computed it in 0.3:
from altar.models import ModelPlugin
from altar_alphagenome import AlphaGenomePlugin


class AlphaGenomeAsIn030(AlphaGenomePlugin):
    reads_reference_genome = True
    equivalence_rules_format = 1
    equivalence_projection = ModelPlugin.equivalence_projection


# `request` holds the configuration and the run inputs, reference included, that the lineage was scored with.
# `identity` is the run identity the lineage was registered with.
descriptor = AlphaGenomeAsIn030().scoring_lineage_descriptor_for(request)
report = await store.attach_lineage_descriptors([(identity, descriptor)])

attach_lineage_descriptors refuses a descriptor that does not reproduce the registered identity, so a wrong request attaches nothing. Then check lineages_without_descriptors("org.kundajelab.altar.model.alphagenome").

A store without lineages holds one identity per model. If its AlphaGenome rows were scored with a reference, the next run has another identity, and altar run stops with an error that offers two ways on: a new [models.<name>] key, or a new store with lineages = true. The old rows cannot be reused.

A resolution frozen with ScoreReusePolicy.exactly(...) keeps reading exactly the lineages it lists.

LegNet labels

LegNet's cell_type and assay name the data a checkpoint was trained on. The runtime never reads them. LegNet's lineage IDs do not change, so a configuration with other labels keeps its own ID. Its equivalence rules now leave the labels out, so lineages that differ only in labels reuse each other's scores. The rules ID is now d3ffce46… (rules format 2). A different checkpoint, training configuration, reference, image or average_reverse is still another model.

LegNet lineages written by 0.3 have descriptors and need nothing. For an older one, attach the descriptor the current plugin derives, LegNetPlugin().scoring_lineage_descriptor_for(request).

The labels are checked where a record says what the checkpoint is:

  • The official weights, the registry entry legnet/wtc11 that altar resources fetch legnet/wtc11 fetches, are the WTC11 LentiMPRA checkpoint. A configuration that labels them otherwise is refused.
  • LegNetConfiguration.from_resource_manifest takes cell_type from the manifest, and raises ValueError if you pass another. In 0.3, the argument overrode the manifest.

Altar cannot check a custom checkpoint's labels. Package it with the runtime's legnet-resources package --cell-type, naming the cell type its publisher states, and use a label such as unknown when the training cell type is not known. See LegNet.

Other changes

None of these needs action.

New commands and spec sections

  • altar resources reads registries: indexes of entries, each a model's saved configuration and the files it names, or a genome's FASTA. Installed packages ship the genome and legnet registries, and a spec's [registries] or ALTAR_REGISTRIES adds others. altar resources fetch downloads an entry into the cache and checks every file's digest; altar resources fetch --spec SPEC fetches everything a spec names. A spec names a model entry with [models.<name>] resources = "legnet/wtc11", which also gives the model's type, and a genome with [variants] reference = "hg38". A run never downloads: an entry that is not fetched stops it with the command to run. resources may be a list, which scores one model per entry. See Find and fetch models and genomes. It is provisional.
  • Genomes are named <assembly>/<authority>/<product>, such as grch38/ucsc/hg38 or grch38/ncbi/no-alt-analysis-set; hg38, hg19, mm10 and mm39 are aliases. See The genomes.
  • A registry that is not on your disk can only make a fetch read public https://, gs://, s3:// and hf:// files, never local files or hosts on your own network unless you set allow_private_hosts. Cloud files are read without your credentials unless the registry names a connection. A download stops at the size its entry gives, or at 64 GB (ALTAR_FETCH_MAX_BYTES).
  • A spec can define several backends as [backends.<name>]. Each model then names its backend, and any model can pick a GPU pool, for every stage or per stage. The local backend's pools option gives each pool its own devices. Neither choice is part of a model's identity. See Run different models on different machines or pools.
  • While tasks run, altar run, altar prepare and altar interpret print how many variants are left, the tasks submitted and the time limit, then how many tasks are running, waiting and finished.
  • A spec with models and no [store] keeps scores in SQLite at <cache_dir>/scores.sqlite. In 0.3 it was refused.
  • altar run reads a cohort TSV with tabs or runs of spaces between fields, and skips blank lines.
  • altar prepare runs a model's preparation from a spec and publishes the prepared files. A later altar run checks those files against the scoring configuration. For ChromBPNet it compares the published manifest with the configuration's weights, peaks, strand policy and preprocessing version, and turns on verify_preprocessing for the run. An explicit verify_preprocessing = false is kept, with a warning. See Prepare model resources. altar prepare is provisional.
  • altar interpret and altar interpretations, with the [interpretations.<name>] and [interpretation_store] spec sections, run and list interpretation methods: ChromBPNet's deepshap_counts and finemo_hits, and Cherimoya's ism_counts. See Interpret variants. Interpretation is provisional.
  • altar-gpnstar-build --overwrite replaces a previous build once the new one is complete.

Exports and run summaries

  • Each export records what produced it. A Parquet export carries it in the file's key-value metadata, under altar.provenance, and a TSV export writes _provenance.json beside the table. _run_summary.json and RunSummary gain a provenance field. See Read a run's results.
  • altar run writes _columns.tsv beside every wide export, one row per column, naming the model, plugin and result schema each column comes from. The wide header is unchanged. See Find a model's columns.
  • [export] layout = "long" writes one table per result schema, plus models and variants tables, in place of one wide table. See Long format for many models. It is provisional.
  • augment_run_export writes an augmented VCF from a Parquet export (provisional). Six source bindings declare their INFO fields for it. See Write an augmented VCF.

Python API

Most changes to stable names are additions. The exceptions are the max_ticks and max_ticks_per_wave parameters removed from run_scoring, score_cohort and run_model_workflow, and the new fields in the middle of ReconcileResult, all listed under Python code and extensions. No stable name was removed or renamed.

  • altar.interpretation and altar.registries are new modules, and every name in them is provisional.
  • run_scoring, score_cohort and run_model_workflow take progress, a callable that receives the progress lines. run_reconcile_loop takes on_tick.
  • ingest_scoring_results takes an optional plan=, so an application that runs the containers itself can ingest their outputs under the plan it saved at submission instead of a rebuilt one. See Ingest outputs you collected.
  • ScoreStore, DetailStore, ArtifactStore and AnnotationSource gain close() and work in a with block. altar run closes every store and source it builds, once, however the run ends. Sources passed in by the caller stay open.
  • LocalStorage and FsspecStorage take fsync=True to sync each new file and its directory to disk. It is off by default.
  • LocalDockerExecutionBackend takes pools. ExecutionBackend.gpu_pools() names the pools a backend defines; it returns None by default, which accepts any pool.
  • run_model_workflow(collect_completed=True) keeps the outputs of the stages that finished before a failure, and WorkflowExecutionError names them (provisional).
  • New provisional names: ModelPlugin.reads_reference_genome and ModelPlugin.prepared_run_inputs; ExecutionBackend.adopts_active_tasks, clears_outputs_on_fresh_start, writes_output_in_place and gpu_pools, with require_gpu_pool and UnknownPoolError; altar.run.prepare, interpret and their summaries; wide_column_manifest, write_column_manifest, write_long_export and the other column-manifest and long-export names in altar.results; AnnotationSource.option_descriptions; altar.sources.add_release_argument; altar.variants.AnnotationInfoField, check_annotation_info_fields and augment_run_export; check_registry and assert_valid_registry in altar.testing. Provisional names lists each one.

A binding whose model reads no local sequence can set reads_reference_genome = False. It then keeps ModelRunInputs as its run-input base, and overrides equivalence_projection as AlphaGenome does if lineages that recorded a reference exist. The scorer conformance suites check the declaration.

Next

  • Maturity labels: what Stable, Provisional and Preview mean, and which names are provisional.