Fix a run that stops or warns¶
These are the messages altar run prints when it stops or warns, and what to do about each. Most also apply to
altar prepare and altar interpret, which print the same messages under their own names.
Read what the run reports¶
altar run prints its progress on standard error, one line per stage, and the run summary as JSON on standard
output. A progress line starts with the spec section it belongs to, such as [variants]. A problem starts with
altar run: warning: when the run went on past it, or altar run: error: when it stopped the run. An error says
what went wrong and, for a problem in the spec, names the section to change:
altar run: error: [sources.gnomad] type 'gnomd' is not an installed annotation source; did you mean 'gnomad'? Installed: alphamissense, bigquery, clinvar, gnomad, gpnstar, regions, spliceai, sqlite, variant_gene_links.
A spec that does not parse lists one problem per indented line, each with the path of the key:
altar run: error: invalid run spec:
run.job_id: String should match pattern '^[A-Za-z0-9][A-Za-z0-9._-]*$'
variants.formt: Extra inputs are not permitted
exports: Extra inputs are not permitted
The exit status says what kind of stop it was:
| Status | Meaning | Next step |
|---|---|---|
0 |
The run finished. Warnings may still have been printed. | Read The run warns. |
2 |
The run refused the spec, a file, or a missing extra or credential. | Change what the message names; see The run refuses the spec. |
1 |
The run failed partway. | Rerun with -v; see The run fails. |
130 |
You interrupted it with Ctrl-C. The export was not written. | Run the same command again. |
Exit statuses lists the statuses of every command.
Three flags help:
altar run --validate spec.toml(Provisional) makes every check that needs no variant, then stops. It reads no variant and writes no file, so it is the fastest way to find a spec problem.-v(--verbose) prints debug lines, the full traceback when the run stops, and the time on every line. A failure the run did not foresee ends withRun again with -v to see where it stopped.-q(--quiet) prints only warnings and errors.
The run refuses the spec¶
A refusal exits with status 2. Most refusals happen before the cohort is read.
A name, key or type is misspelled¶
An unknown section or key is refused with its path, as above. A misspelled type gets the installed name it is
closest to. A type is matched exactly, so type = "legnet" is refused with did you mean 'LEGNET'?. A model's
configuration key gets the same help:
altar run: error: models.legnet-wtc11.configuration: unknown option 'average_revers'; did you mean 'average_reverse'? Valid options: weights, training_config, cell_type, assay, average_reverse, image.
altar models <type> and altar sources <type> list every key a model or a source takes. The
run spec reference lists every section and key.
A section is missing¶
altar run: error: the run spec declares no [models.<name>] and no [sources.<name>], so it has nothing to score or annotate; add a model to score the cohort, or a source to annotate it
altar run: error: [models.legnet-wtc11] (LEGNET) runs in containers, so the spec needs a [backend] to run them; add [backend] with type = "local" to run them with Docker on this machine
altar run: error: models.legnet-wtc11.run_inputs: genome is required; set [variants] reference_fasta, since this model reads the reference genome
altar run: error: models.sei.output selects artifact outputs (chromatin_profile_effects), but the spec declares no [artifact_store] to store them; add one (for example type = "storage")
Add the section the message names. A model whose prioritization reads an annotation column, such as ChromBPNet's
region_type, also needs the source that supplies it, and the message names the [sources.<name>] section to add.
A file is not where the spec says¶
A relative path in a spec resolves against the directory you start altar in, not the spec's directory. Start
the command from the directory the spec's paths are relative to, or write absolute paths. The same rule places
the run's directory, the cache (.altar unless [run] cache_dir or ALTAR_CACHE_DIR names another), a [store]
file and the local backend's data_root: a run started from another directory writes them in a new place and
reuses no stored score.
A VCF whose name does not end in .vcf, .vcf.gz or .vcf.bgz would be read as a TSV, and is refused:
altar run: error: variants.path: cohort.txt is a VCF, but its name does not end in .vcf, .vcf.gz or .vcf.bgz, so it would be read as a canonical TSV; set [variants] format = "vcf"
A malformed row stops the run with the row it could not read.
The run directory already exists¶
altar run: error: run.work_dir: the run directory defaults to the job_id, and hg38 already exists and holds files another run did not write. Set [run] work_dir to a new directory, or choose another job_id
Without [run] work_dir, a run writes to a directory named after its job_id. When a directory of that name
already holds other files, the run stops before it writes anything. Set work_dir, or rename the job_id. A
directory an earlier run with the same job_id wrote is used again.
The genome builds do not match¶
The run checks every model and source against [run] genome_build before it opens any data, and names every one
that cannot serve it:
altar run: error: [sources.gnomad] serves hg38 only; the run is hg19.
[sources.clinvar] reads its publisher's copy, which is hg38; the run is hg19. Point it at hg19 data with its dataset option (its binding documents how to build one).
Remove [sources.gnomad] or lift the cohort over to hg38; if the cohort is already on hg38, set [run] genome_build = "hg38".
Fix genome_build if it is wrong, remove the sources that serve only another build, or point a source at data on
the run's build. Give a run its variants explains the check.
The reference digest does not match¶
altar run: error: variants.reference_digest: ... Restore the expected FASTA, or update reference_digest only if this reference change is intended; reference-reading models will use a different score identity
The FASTA at reference_fasta is not the file the digest names. Restore it, or update the digest if you meant to
change the reference. Models that read the reference then score the cohort again.
A credential or an extra is missing¶
altar run: error: [models.alphagenome] configuration.credential: the environment variable ALPHAGENOME_API_KEY is not set; export it before running
altar run: error: [sources.phylop]: reading hf:// needs the hf extra: pip install "altar[hf]"
Export the variable the spec names, or install the extra. A Google Cloud error about credentials, permissions or a
missing project or dataset is a refusal too. Its message ends with what to do, such as running
gcloud auth application-default login.
The local backend's data_root is not usable¶
Name a directory you can create and write. Without a data_root, the backend uses backend-data in the cache,
and the message names run.cache_dir. altar prepare also refuses a data_root inside <cache_dir>/staging, or
one that contains it.
Prepared files are missing or stale¶
A model that reads files a preparation step writes, such as ChromBPNet, needs altar prepare first:
altar run: error: [models.k562] reads files a preparation step writes, and .altar/staging does not have them: ... Run `altar prepare SPEC --model k562` with this spec first; https://kundajelab.github.io/altar/integrations/chrombpnet/ describes it
altar run: error: [models.k562] preparation publication is incomplete; run `altar prepare SPEC --model k562` successfully before scoring, even when scores are already cached
altar run: error: [models.k562] the files `altar prepare` published were prepared with ... Run `altar prepare SPEC --model k562` with this spec, then score again
The first means preparation has not run with this cache: start both commands from the same directory, or
give them the same cache_dir. The second means a preparation stopped while it
published its files. The third means the model's configuration changed after preparation. In each case run
altar prepare with the same spec and the same [models.<name>] key, then score again; see
Prepare model resources.
A changed setting meets a store without lineages¶
altar run: error: [models.legnet-wtc11]: stored scores for model 'legnet-wtc11' use incompatible scientific scoring semantics. The store at legnet-run/scores.sqlite keeps no lineages, so it holds one setting per model key. Give the changed model a new key, such as [models.legnet-wtc11-2], or keep scores in a new store with lineages = true, such as [store] options = { path = "scores-lineages.sqlite", lineages = true }
A store created without lineages holds one set of settings per model key, so it cannot keep a changed setting's
scores beside the old ones. Do one of the two things the message names. A lineages option that disagrees with an
existing store file is refused too; remove the option to open the file as it was created.
A store cannot be built¶
altar run: error: store: 'bigquery' cannot be built from the spec's options (BigQueryScoreStore.__init__() missing 1 required positional argument: 'client'); it reaches its backend through a connection, so declare one and name it with store.connection
A bigquery store or detail store needs a gcp connection, named with connection. A connection on a sqlite
store or on local storage is refused. Choose where results are stored covers these
and a SQLite file in a missing directory.
The run fails¶
A failure exits with status 1. It is something the checks could not foresee: a container task that failed, a
lost connection, a task that ran too long. Rerun with -v to see where it stopped. Scores of the batches that
finished before the failure are stored, and the next run reuses them.
A container task fails¶
The message names the failed task, its exit code, a reason such as exit_1 or oom_killed, and the last lines
of the container's log:
altar run: error: ReconcileStalled: reconciliation stalled — group(s) blocked by a failed shard with no on_failed: ...
Read the end of the log in the message. The common causes on the local backend:
| Cause | Sign | Fix |
|---|---|---|
| No usable GPU | RuntimeError: docker run failed (exit 125): docker run --detach ... --gpus 1 ..., followed by Docker's own message |
Install the NVIDIA Container Toolkit, or set [backend] cpu_only = true to run the task on the CPU. |
| The image cannot be pulled | RuntimeError: docker run failed ... with Docker's pull error, after a warning that the image was not found |
Check the network and the image name and digest. The images are built for linux/amd64; see Platform support. |
| The container cannot write its outputs | A task fails, and the log reports a permission error under /mnt/volume |
On Linux the container runs as the owner of data_root. For a root-squashed NFS data_root, set [backend.options] container_user = "uid:gid" to a user that can write it. |
| Out of memory | The reason is oom_killed |
Free memory on the machine. On macOS, raise Docker Desktop's memory limit. |
| The tasks ran too long | altar run: error: [models.<name>] tasks still running after 24h (the [backend] timeout) |
See The tasks ran too long. |
A kubernetes backend declared in a spec needs volume_uri in its options. Without it the run stops with
ValueError: storage and volume_uri select the staged mode together; set both or neither; see
backend.
The tasks ran too long¶
The run waits for each batch of a model, and for each wave of stages of altar prepare and altar interpret, at
most [backend] timeout, 24 hours unless the spec sets it. Tasks still open after that stop the run:
altar run: error: [models.chrombpnet] tasks still running after 24h (the [backend] timeout); 3 tasks remain open, and they were cancelled. Raise [backend] timeout, or lower [models.chrombpnet] max_variants_per_batch so each batch finishes sooner; running the spec again keeps the batches already stored and starts this one over
Raise timeout, for example to "2d", or score fewer variants per batch with the model's
max_variants_per_batch. What happens to the tasks still open depends on the backend:
| Backend | The open tasks | Running the spec again |
|---|---|---|
local |
Stopped and removed (docker rm --force). |
Starts the batch over in new containers. |
kubernetes |
Their Jobs are deleted, and Kubernetes then removes their pods. | Creates the Jobs again, so the batch starts over. |
modal |
Keep running until the task's own time limit, at most Modal's 24 hours, because the backend cannot stop a sandbox. Stop them from the Modal dashboard if you do not want them to finish. | Starts new sandboxes; it does not use the ones still running. |
In every case the batches that finished before the limit are stored, and a rerun scores only the variants still
missing. altar prepare starts the model's preparation over. altar interpret keeps the products of every method
that finished and runs the rest. Only a run that is killed outright, so that it cannot cancel anything, leaves
Kubernetes Jobs running; a rerun with the same job_id then picks up those still pending or running instead of
starting them again.
A spec written for an earlier release may still set [backend] max_ticks, a count of checks. It is refused; set
timeout instead.
A source cannot be reached¶
A source that cannot open its data stops the run before the cohort is read, naming the section:
Check the network, and the credentials of the connection the section names. A run that reads a published copy
from Hugging Face (location = "hf") can slow down or wait on its rate limit; see
Hugging Face rate limits.
An interpretation leaves variants without products¶
altar interpret exits with status 1 when an operation leaves variants without products:
altar interpret: error: contributions.deepshap_counts left variants without products; see the summary's failure and skipped entries, and run the same command again to retry them
The methods before the failed one stay stored. Run the same command again to retry the missing variants.
The run warns¶
A warning starts with altar run: warning:. It does not stop the run, and the exit status is 0. -q keeps
warnings.
Variants a lookup may miss¶
The run ends with a warning for each kind of variant an annotation source may miss or read at the wrong place: what it is, with one example; why; and what to do. Every variant is still in the export:
altar run: warning: annotation sources may miss 1 indel: chr7:117559591:TCTT:T
It was trimmed but not left-aligned, because [variants] names no reference genome; gnomAD and ClinVar store indels left-aligned.
Set [variants] reference or reference_fasta to left-align it. Every variant is still in the export.
| Reason | What to do |
|---|---|
indel_not_left_aligned |
Set [variants] reference_fasta. |
reference_mismatch |
Check the REF alleles, and that [run] genome_build is the cohort's build. |
contig_not_in_reference |
Use a FASTA of the cohort's build that holds every contig the cohort uses. |
contig_not_primary |
Nothing, if expected: the published datasets cover chr1–22, X, Y and M only. |
The run summary's potential_misses holds the same counts. Annotate variants with
evidence explains each kind.
VCF alleles were skipped¶
altar run: warning: skipped 3 VCF ALT alleles that name no sequence a variant key can hold: 2 symbolic (<...>), 1 spanning deletion (*) (e.g. chr1:100:A:<DEL>)
The export does not have them.
Symbolic alleles, breakends, *, ., an ALT equal to REF and IUPAC codes have no sequence to score or look up.
The run summary's dropped_alleles counts them by reason.
An image or Docker was not found¶
altar run: warning: [models.legnet-wtc11] image ... is not on this machine, and its registry did not return it (...). The run will fail when it starts the container unless the image can be pulled by then; check the image name and digest
altar run: warning: [backend] the local backend runs containers with docker, which is not on PATH
The run continues, since the registry may be out of reach only for now. Pull the image, or install Docker, before the run reaches that model.
Other warnings¶
| Warning | Meaning |
|---|---|
[export] format = "csv" is a deprecated name for "tsv" |
Write format = "tsv", or leave format out for Parquet. |
the TSV holds ... as JSON text, which can exceed a spreadsheet's cell limit |
Leave [export] format out to keep list and record columns as lists in Parquet. |
The run scored everything again¶
A model's line says how many variants were already scored:
[models.legnet-wtc11] 5 candidates, 1 set aside (1 deletion; listed in legnet-demo/staging/jobs/legnet-demo/variants/legnet-wtc11/ineligible.tsv), 4 already scored, 0 scored in 0 batches, in 0.0s
When a rerun scores variants you expected to be reused, check what changed:
- the model key, the
<name>of[models.<name>], under which the store keeps the scores; - a setting that changes the scores, which starts a new lineage beside the old one;
- the reference FASTA, for a model that reads it;
- the cache, for a spec without
[store]: another starting directory,[run] cache_dirorALTAR_CACHE_DIRfinds anotherscores.sqlite; - the
[store]path, or the directory the command starts in when the path is relative; - a
[store]without apath, which keeps scores in memory for one run only; - a
[detail_store]that moved, for a model with detail tables.
Keep scores in a SQLite file shows a store that later runs reuse.
A variant counts as scored when the store holds its row for the model's key and settings, and its detail rows and artifacts when the model has them. Score reuse and provenance explains when a score is reused, and Tell exports apart shows which settings an export's scores came from.
A model set variants aside¶
The same line counts the variants the model set aside, by reason, and names the file that lists them. A model
scores only the variant classes and allele lengths its binding accepts; altar models <type> lists them. A
variant whose REF does not match the reference, or whose contig the reference lacks, is set aside by every model.
Get more help¶
Run the command again with -v and keep its standard error. Keep the run summary on standard output, or
_run_summary.json in the export directory. Get help says where to report a problem.