Upgrade from 0.3 to 0.4¶
Most changes from Altar 0.3 to 0.4 need no action. The ones that do affect run specs that set
[backend] max_ticks, work_dir or reference_digest, scripts that read the commands' standard error, stores
that hold AlphaGenome or LegNet scores, ChromBPNet interpretation results that your own application caches, scripts
that call the dataset builders, code that calls the scoring drivers with max_ticks, and custom execution
backends.
Before you upgrade¶
- Upgrade
altar,altar-identityand every binding together. Each 0.4 binding requiresaltar>=0.4.0,<0.5. - Altar needs Python 3.12 or newer; 3.12 and 3.13 are tested. See Platform support.
- Score lineages are the same in 0.3 and 0.4, with one exception: an AlphaGenome run given a reference genome now has the lineage of the same run without one. AlphaGenome and the reference genome explains what to do. Every other stored score stays a cache hit after the upgrade.
Changes that need action¶
Run specs¶
| Change | Who is affected | What to do |
|---|---|---|
[backend] max_ticks, a count of checks, is replaced by timeout, a time limit for each batch of a model or each wave of workflow stages. It takes seconds or text such as "90m", "24h" or "2d", and defaults to 24 hours. A spec that sets max_ticks is refused |
Specs that set max_ticks |
Replace it with timeout. See The tasks ran too long for what happens to open tasks at the limit on each backend |
[backend] poll_interval_s defaults to 5 seconds instead of 1 |
Specs that rely on the default | Nothing, unless you want quicker checks: set poll_interval_s |
A run keeps its files in two directories. [run] work_dir, now optional with a default of ./<job_id>, holds only this run's files and the export. A new [run] cache_dir, $ALTAR_CACHE_DIR or ./.altar by default, holds what later runs reuse: prepared files under staging/, the default artifact store under artifacts/, interpretations and fetched entries. In 0.3, prepared files and the default artifact store were under work_dir |
Specs whose models read prepared files, such as ChromBPNet, or keep artifacts in the default store | Set cache_dir to the old work_dir to keep using what is there, or run altar prepare again. See the run section |
Without work_dir, a run refuses to start when ./<job_id> already holds files another run did not write |
Specs that leave out work_dir |
Set work_dir, or choose another job_id. See The run directory already exists |
The local backend's default data_root is backend-data in the cache. ALTAR_LOCAL_DATA_ROOT is no longer read, and altar.plugins.execution.DATA_ROOT_ENV is removed. Without a data_root, LocalDockerExecutionBackend uses $ALTAR_CACHE_DIR/backend-data, or ./.altar/backend-data, in place of ./.altar-local-data |
Anyone who set ALTAR_LOCAL_DATA_ROOT, or reads files under .altar-local-data |
Set [backend] options = { data_root = "..." }, or pass data_root= in Python |
A task that asks for a GPU pool its backend does not define is refused before anything is submitted, with UnknownPoolError. The local backend defines gpu, or the names in its new pools option; kubernetes defines gpu and the names in pool_node_selectors and max_slots; modal defines the keys of pool_accelerators |
Specs and bindings that ask for a pool name a backend does not list | Add the pool to the backend's options, or ask for one it defines. See GPU pools |
The modal backend no longer upper-cases a pool name it does not map into an accelerator name |
Modal users whose tasks ask for a pool missing from pool_accelerators, such as a100 |
Map the pool in pool_accelerators, such as { gpu = "A10G", a100 = "A100" }. Its keys replace the default, so keep gpu if your tasks use it |
[variants] reference_digest is checked against the FASTA on every run, before any lookup. A mismatch stops the run |
Anyone whose spec sets reference_digest |
Make sure the digest is the FASTA's. Change it only if you meant to change the reference: models that read the reference then score under a new lineage |
The first run with reference_digest writes <fasta>.sha256-verified beside the FASTA |
Anyone whose spec sets reference_digest |
Nothing. Later runs read this record instead of hashing an unchanged file. If the run cannot write in the FASTA's directory, it hashes the file on every run |
A spec that sets reference_digest without reference_fasta is refused |
Specs that set only the digest | Add reference_fasta, or remove the digest |
| A run whose model needs an annotation source for its prioritization rule now stops before scoring if the spec lacks that source. It used to fail at export | Specs that score such a model without its source | Add the [sources.<name>] section the error names |
Command output¶
| Change | Who is affected | What to do |
|---|---|---|
Every altar command prints standard error in one format. A progress line has no altar run: prefix and starts with its spec section, such as [models.<name>]. A problem starts with altar <command>: warning: or altar <command>: error:, and continues on lines indented two spaces. Each variant a lookup may miss is its own warning. Counts agree in number instead of (s), and altar pull prints sizes in decimal units. Standard output and its JSON are unchanged |
Scripts that read the commands' standard error | Match the new lines, or read the JSON on standard output. See the command-line reference |
Stored results¶
| Change | Who is affected | What to do |
|---|---|---|
| An AlphaGenome run given a reference has the lineage of the run without one | Stores that hold AlphaGenome scores from runs with reference_fasta |
See AlphaGenome and the reference genome |
LegNet lineages that differ only in cell_type or assay reuse each other's scores |
Anyone who kept such lineages apart on purpose | Nothing; their scores are identical. Read with ScoreReusePolicy.exactly(...) to keep a fixed set |
LegNetConfiguration.from_resource_manifest refuses a cell_type that differs from the manifest's, and the official LegNet weights must be labelled WTC11 and LentiMPRA |
Callers that relabelled a checkpoint | Leave cell_type out, or correct the manifest. See LegNet labels |
ChromBPNet's interpretation runs the seeded DeepSHAP method, deepshap_counts 2.0.0, on its own pinned image, and its motif stage runs riyavsinha/altar-finemo, which runs as a non-root user. build_interpretation_plan uses them too, so its interpretation run IDs change |
Applications that cache ChromBPNet interpretation results by run ID | Start a new interpretation run after the upgrade. Results stored under the old IDs are not reused, and the two methods are not treated as the same. Score lineages do not change |
Python code and extensions¶
| Change | Who is affected | What to do |
|---|---|---|
run_scoring and score_cohort take timeout_s (default 86,400 seconds) in place of max_ticks, and run_model_workflow takes it in place of max_ticks_per_wave. Their poll_interval_s defaults to 5 seconds instead of 1. Tasks still open at the limit are cancelled and TimeoutError is raised |
Code that passes max_ticks or max_ticks_per_wave |
Pass timeout_s, in seconds. run_reconcile_loop keeps max_ticks and also takes timeout_s |
ReconcileResult has two new fields, running and pending, after failed |
Code that builds a ReconcileResult with positional arguments |
Pass its fields by name |
RunPlan has a new cache_dir field after work_dir |
Code that builds a RunPlan with positional arguments |
Pass its fields by name. RunPlan is Provisional |
The kubernetes backend reports an active Job whose pod is not ready yet as PENDING, not RUNNING. A cluster that does not report status.ready, before Kubernetes 1.24, still reports every active pod as RUNNING |
Code that reads task states from KubernetesExecutionBackend |
Treat PENDING as a task that has not started: it may be scheduling or pulling its image |
Deprecated builder flags: altar-popfreq-build --version, altar-clinvar-build --release-date, and --source-release on altar-alphamissense-build and altar-spliceai-build. altar-e2g-atlas-ingest takes --input and --database in place of two positional arguments |
Scripts that build source datasets | Use --release, and --input … --database …. The old forms still work in 0.4 and print a warning |
run_scoring refuses a backend that adopts running tasks (adopts_active_tasks) without declaring clears_outputs_on_fresh_start. A subclass that overrides submit resets that declaration to False |
Custom execution backends that adopt tasks, and subclasses of KubernetesExecutionBackend that override submit |
Clear each output before a fresh attempt starts, and declare clears_outputs_on_fresh_start = True. See Clear outputs an earlier run left |
New conformance tests: the execution-backend suite checks output clearing on a fresh start, an adopted task and a replaced task, and the refusal of an undefined GPU pool; the storage suite checks failed and interrupted writes; the store and source suites check that close() can be called twice |
Authors of backends, storage adapters, stores and sources | Run the suites. A backend that declares adopts_active_tasks or clears_outputs_on_fresh_start implements the suite's write_task_output and task_output hooks. A backend that defines its GPU pools implements gpu_pools(); see Declare your GPU pools |
LocalStorage and FsspecStorage write a local file through a temporary file and a rename. A destination that is a symbolic link is replaced, not written through |
Code that writes to a symbolic link through Altar's storage | Write to the link's target path instead |
AlphaGenome and the reference genome¶
AlphaGenome never reads a local reference: the hosted API supplies its own sequence. In 0.3, a run given
[variants] reference_fasta still recorded the FASTA's digest in AlphaGenome's lineage, so adding a FASTA for
left-alignment made every stored AlphaGenome score a cache miss. In 0.4, AlphaGenome's run inputs carry no
reference, and its lineage records the genome build only. A run that carried a reference therefore moves to the
lineage of the same run without one. For the binding's human_defaults fixture:
| Run | Lineage ID in 0.3 | Lineage ID in 0.4 |
|---|---|---|
| No reference | d92e7b1c… |
d92e7b1c… |
Reference with digest sha256:aaaa…, at any path |
ad43d218… |
d92e7b1c… |
Reference with digest sha256:bbbb… |
657bc3b8… |
d92e7b1c… |
| Reference without a digest | 3bf3b854… |
d92e7b1c… |
AlphaGenome's run identity no longer carries genome_hash. AlphaGenome's equivalence rules drop the reference
from an older lineage's descriptor, so scores stored under a lineage that recorded a reference stay reachable in a
store that keeps lineages. The rules ID is now fbf5b668… (rules format 2).
- Lineages written by 0.3 have descriptors. Nothing to do: the next resolution lists the older lineage as
equivalent, and its rows are hits. - Lineages written before 0.3 have no descriptor. Attach one, rebuilt as AlphaGenome computed it in 0.3:
from altar.models import ModelPlugin
from altar_alphagenome import AlphaGenomePlugin
class AlphaGenomeAsIn030(AlphaGenomePlugin):
reads_reference_genome = True
equivalence_rules_format = 1
equivalence_projection = ModelPlugin.equivalence_projection
# `request` holds the configuration and the run inputs, reference included, that the lineage was scored with.
# `identity` is the run identity the lineage was registered with.
descriptor = AlphaGenomeAsIn030().scoring_lineage_descriptor_for(request)
report = await store.attach_lineage_descriptors([(identity, descriptor)])
attach_lineage_descriptors refuses a descriptor that does not reproduce the registered identity, so a wrong
request attaches nothing. Then check lineages_without_descriptors("org.kundajelab.altar.model.alphagenome").
A store without lineages holds one identity per model. If its AlphaGenome rows were scored with a reference, the
next run has another identity, and altar run stops with an error that offers two ways on: a new
[models.<name>] key, or a new store with lineages = true. The old rows cannot be reused.
A resolution frozen with ScoreReusePolicy.exactly(...) keeps reading exactly the lineages it lists.
LegNet labels¶
LegNet's cell_type and assay name the data a checkpoint was trained on. The runtime never reads them. LegNet's
lineage IDs do not change, so a configuration with other labels keeps its own ID. Its equivalence rules now leave
the labels out, so lineages that differ only in labels reuse each other's scores. The rules ID is now d3ffce46…
(rules format 2). A different checkpoint, training configuration, reference, image or average_reverse is still
another model.
LegNet lineages written by 0.3 have descriptors and need nothing. For an older one, attach the descriptor the
current plugin derives, LegNetPlugin().scoring_lineage_descriptor_for(request).
The labels are checked where a record says what the checkpoint is:
- The official weights, the registry entry
legnet/wtc11thataltar resources fetch legnet/wtc11fetches, are the WTC11 LentiMPRA checkpoint. A configuration that labels them otherwise is refused. LegNetConfiguration.from_resource_manifesttakescell_typefrom the manifest, and raisesValueErrorif you pass another. In 0.3, the argument overrode the manifest.
Altar cannot check a custom checkpoint's labels. Package it with the runtime's legnet-resources package
--cell-type, naming the cell type its publisher states, and use a label such as unknown when the training cell
type is not known. See LegNet.
Other changes¶
None of these needs action.
New commands and spec sections¶
altar resourcesreads registries: indexes of entries, each a model's saved configuration and the files it names, or a genome's FASTA. Installed packages ship thegenomeandlegnetregistries, and a spec's[registries]orALTAR_REGISTRIESadds others.altar resources fetchdownloads an entry into the cache and checks every file's digest;altar resources fetch --spec SPECfetches everything a spec names. A spec names a model entry with[models.<name>] resources = "legnet/wtc11", which also gives the model's type, and a genome with[variants] reference = "hg38". A run never downloads: an entry that is not fetched stops it with the command to run.resourcesmay be a list, which scores one model per entry. See Find and fetch models and genomes. It is provisional.- Genomes are named
<assembly>/<authority>/<product>, such asgrch38/ucsc/hg38orgrch38/ncbi/no-alt-analysis-set;hg38,hg19,mm10andmm39are aliases. See The genomes. - A registry that is not on your disk can only make a fetch read public
https://,gs://,s3://andhf://files, never local files or hosts on your own network unless you setallow_private_hosts. Cloud files are read without your credentials unless the registry names a connection. A download stops at the size its entry gives, or at 64 GB (ALTAR_FETCH_MAX_BYTES). - A spec can define several backends as
[backends.<name>]. Each model then names itsbackend, and any model can pick a GPUpool, for every stage or per stage. Thelocalbackend'spoolsoption gives each pool its own devices. Neither choice is part of a model's identity. See Run different models on different machines or pools. - While tasks run,
altar run,altar prepareandaltar interpretprint how many variants are left, the tasks submitted and the time limit, then how many tasks are running, waiting and finished. - A spec with models and no
[store]keeps scores in SQLite at<cache_dir>/scores.sqlite. In 0.3 it was refused. altar runreads a cohort TSV with tabs or runs of spaces between fields, and skips blank lines.altar prepareruns a model's preparation from a spec and publishes the prepared files. A lateraltar runchecks those files against the scoring configuration. For ChromBPNet it compares the published manifest with the configuration's weights, peaks, strand policy and preprocessing version, and turns onverify_preprocessingfor the run. An explicitverify_preprocessing = falseis kept, with a warning. See Prepare model resources.altar prepareis provisional.altar interpretandaltar interpretations, with the[interpretations.<name>]and[interpretation_store]spec sections, run and list interpretation methods: ChromBPNet'sdeepshap_countsandfinemo_hits, and Cherimoya'sism_counts. See Interpret variants. Interpretation is provisional.altar-gpnstar-build --overwritereplaces a previous build once the new one is complete.
Exports and run summaries¶
- Each export records what produced it. A Parquet export carries it in the file's key-value metadata, under
altar.provenance, and a TSV export writes_provenance.jsonbeside the table._run_summary.jsonandRunSummarygain aprovenancefield. See Read a run's results. altar runwrites_columns.tsvbeside every wide export, one row per column, naming the model, plugin and result schema each column comes from. The wide header is unchanged. See Find a model's columns.[export] layout = "long"writes one table per result schema, plusmodelsandvariantstables, in place of one wide table. See Long format for many models. It is provisional.augment_run_exportwrites an augmented VCF from a Parquet export (provisional). Six source bindings declare their INFO fields for it. See Write an augmented VCF.
Python API¶
Most changes to stable names are additions. The exceptions are the max_ticks and max_ticks_per_wave
parameters removed from run_scoring, score_cohort and run_model_workflow, and the new fields in the middle of
ReconcileResult, all listed under Python code and extensions. No stable name was
removed or renamed.
altar.interpretationandaltar.registriesare new modules, and every name in them is provisional.run_scoring,score_cohortandrun_model_workflowtakeprogress, a callable that receives the progress lines.run_reconcile_looptakeson_tick.ingest_scoring_resultstakes an optionalplan=, so an application that runs the containers itself can ingest their outputs under the plan it saved at submission instead of a rebuilt one. See Ingest outputs you collected.ScoreStore,DetailStore,ArtifactStoreandAnnotationSourcegainclose()and work in awithblock.altar runcloses every store and source it builds, once, however the run ends. Sources passed in by the caller stay open.LocalStorageandFsspecStoragetakefsync=Trueto sync each new file and its directory to disk. It is off by default.LocalDockerExecutionBackendtakespools.ExecutionBackend.gpu_pools()names the pools a backend defines; it returnsNoneby default, which accepts any pool.run_model_workflow(collect_completed=True)keeps the outputs of the stages that finished before a failure, andWorkflowExecutionErrornames them (provisional).- New provisional names:
ModelPlugin.reads_reference_genomeandModelPlugin.prepared_run_inputs;ExecutionBackend.adopts_active_tasks,clears_outputs_on_fresh_start,writes_output_in_placeandgpu_pools, withrequire_gpu_poolandUnknownPoolError;altar.run.prepare,interpretand their summaries;wide_column_manifest,write_column_manifest,write_long_exportand the other column-manifest and long-export names inaltar.results;AnnotationSource.option_descriptions;altar.sources.add_release_argument;altar.variants.AnnotationInfoField,check_annotation_info_fieldsandaugment_run_export;check_registryandassert_valid_registryinaltar.testing. Provisional names lists each one.
A binding whose model reads no local sequence can set reads_reference_genome = False. It then keeps
ModelRunInputs as its run-input base, and overrides equivalence_projection as AlphaGenome does if lineages that
recorded a reference exist. The scorer conformance suites check the declaration.
Next¶
- Maturity labels: what Stable, Provisional and Preview mean, and which names are provisional.