Find and fetch models and genomes¶
A run names its models and its reference genome by entries of a registry. A registry is an index of models and
genomes that anyone can host, and each entry names its files at fixed addresses, each with a recorded SHA-256
digest. altar resources finds entries, altar resources fetch downloads them and checks every file, and a run
spec then names the entry instead of giving paths.
altar resources, the registry format and the run-spec keys on this page are
Provisional.
See the registries¶
The listing has one row per registry: its name, how many entries it lists, where it comes from, and what it holds.
NAME ENTRIES FROM DESCRIPTION
genome 7 altar Reference genome FASTAs, named in a run spec as [variants] reference
legnet 1 altar-legnet Published LegNet checkpoints, which predict MPRA regulatory activity
Altar itself ships genome, with the reference genomes. Each model package you install can add its own; this
legnet registry comes from altar-legnet. Add a registry shows how to add one someone else
publishes.
Find an entry¶
Name a registry to list its entries:
ID ALIAS BUILD AUTHORITY ACCESSION PRODUCT SIZE FETCHED
grcm39/ucsc/mm39 mm39 mm39 UCSC GCA_000001635.9 mm39 2.8 GB no
grcm38/ucsc/mm10 mm10 mm10 UCSC GCA_000001635.2 mm10 2.8 GB no
grcm38/encode/mm10-no-alt-analysis-set - mm10 ENCODE - mm10_no_alt_analysis_set_ENCODE 2.8 GB no
grcm39/ucsc/mm39
61 sequences: the chromosomes, random and chrUn contigs, no alt or fix contigs; soft-masked
grcm38/ucsc/mm10
Initial GRCm38 release, 66 sequences with no alt or fix patches; soft-masked
grcm38/encode/mm10-no-alt-analysis-set
mm10.fa's 66 sequences, with chrY's pseudoautosomal region hard-masked with N; soft-masked
A genome's row leads with its id, <assembly>/<authority>/<product>: GRC's name for the assembly, who publishes
the file, and the publisher's own name for it. Then come its alias, if it has one, and who publishes it, followed
by a line on what sets each file apart. A model's row leads with the entry's name, then its id. You copy the id
into commands and specs as <registry>/<id>, such as genome/grcm38/ucsc/mm10 or legnet/wtc11; an id may hold
/ itself. BUILD is the genome the entry is on: a genome's own, or the one a model was trained on.
--where KEY=VALUEkeeps the entries whoseKEYisVALUE, ignoring case. The keys arename,id,type,genome_build, and what a registry describes its entries by:assay,cell_types,organs,systems,life_stages,sex,organismandproject. An ontology term matches its id or its name, so--where cell_types=K562and--where cell_types=EFO:0002067find the same entries. Repeat--whereto require each one.--search WORDSkeeps the entries with a word of their name, id or description terms that starts with each word you give:--search dnase k562.
Name one entry to see everything its registry says about it: its files with their digests and addresses, the genome it was trained on, its metadata, how to cite it, its terms, and whether you have fetched it.
Fetch one¶
A genome needs no genome/: its id, such as grch38/ucsc/hg38, or its alias, such as hg38, names it, and so
does any name whose first part is not a registry's. The command downloads each file the entry names, unpacks
it if the publisher compressed it, and checks its digest. It prints a line per file and ends with the run-spec
lines that name the entry. A genome takes a few minutes, and also gets an index, checked against its contigs.
Some entries take files from an archive a publisher released, such as ENCODE's model tarballs. The fetch downloads
each archive once, extracts the files the entry needs and checks each one, then deletes the archive. Pass
--keep-archives to keep it.
The files go into resources/sha256/ in the cache, .altar/ in the working directory, the same cache a run keeps
its scores in. Each file is kept under its digest, so one that two entries name is stored once. To keep them
somewhere else, set ALTAR_CACHE_DIR, or pass --cache-dir to the command:
A run looks for fetched files in its cache: [run] cache_dir, or ALTAR_CACHE_DIR, or ./.altar. A spec that
sets cache_dir needs the files fetched there too, with --cache-dir and the same directory.
Each file gets a verification record beside it. Running the command again skips every file whose record still matches it, so an interrupted fetch resumes. A file whose digest does not match, because its publisher replaced it at the same address, is deleted, and the command stops with exit status 2 and names the entry, the address and both digests. A run never downloads anything.
Name it in a run spec¶
A fetched genome goes in [variants] reference, and a fetched model in [models.<name>] resources:
The entry gives the model's type, LEGNET, and its whole configuration. A field under
[models.legnet-wtc11.configuration], such as average_reverse = false, changes the entry's value. The run reads
the files from the cache, and takes the genome's digest from its entry, so it does not hash 3 GB on every run. The
genome must be on [run] genome_build.
resources = "legnet/wtc11@sha256:<hex>" pins the entry: the run reads exactly that entry document, even after
the registry lists a newer one. altar resources legnet/wtc11 prints the entry's digest.
If an entry has not been fetched, altar run and altar run --validate stop with exit status 2 and print the
command to run:
altar run: error: variants.reference: genome/grch38/ucsc/hg38 has not been fetched into .altar; run `altar resources fetch genome/hg38` first
A run records each model's registry, the registry's address, the entry's id and the entry's digest in its summary and its export's provenance.
Score with many models¶
To score with a whole compendium, such as hundreds of ChromBPNet models, list the entries in one section:
[models.compendium]
resources = [
"lab/ENCSR685ZMP",
"lab/ENCSR296UHQ",
"lab/ENCSR000EOT",
]
max_variants_per_batch = 50000
The section becomes one model per entry, in the list's order. Every other key of the section, such as
configuration, output, backend, pool or max_variants_per_batch, applies to each of them. Entries may come
from different registries. A type must be every entry's type, and model_name is refused, since each model is
named by its key.
Each model's key is the entry's registry and id joined by -, in lowercase, with each / of the id written as
- and each . as _: lab/ENCSR685ZMP is the model lab-encsr685zmp, lab/v1.2 is lab-v1_2, and
lab/k562/dnase is lab-k562-dnase. Two entries that would get one key, such as lab/a/b and lab/a-b, cannot be
listed in one run; name one in a section of its own. The key never comes from the
entry's display name, so a registry can rename an entry without changing it. It does not depend on the section
either: the list above scores the same models, and reuses the same stored scores, as one section per entry would:
So you can move an entry from one section to another, or from its own section into a list, and keep its scores. A
spec that names one entry twice, or two entries whose keys are the same, such as lab/v1.2 and lab/v1_2, is
refused, and so is a list entry whose key is another section's.
Fetch every entry a spec names, its [variants] reference too, with one command:
Before it downloads anything, it says how much there is. For a spec that lists 1,500 models and names a genome:
A fetch larger than 20 GB asks you to confirm it first, and so does one with an entry whose registry does not
give its size; --yes confirms it, and without a terminal the command stops and says the size, or how many
entries have none. Set ALTAR_FETCH_CONFIRM_BYTES to a number of bytes to change the threshold.
--jobs N fetches up to N entries at once, 4 by default.
altar run --validate checks the list without downloading anything. When some entries are missing, it says how
many, the size still to fetch and the command to run:
altar run: error: models.compendium.resources: 301 of 1,500 entries have not been fetched into .altar (45.2 GB to fetch), such as lab/ENCSR000EOT; run `altar resources fetch --spec score.toml` first, which fetches every entry the spec names
When every entry is fetched, the check prints one line for the section, with its count, in place of a line per model. A run prints a line before the section's first model with how many it holds, one line per model as it scores, with its place in the list, and a line after the last model with the section's totals. The run summary lists the section with its count and its models' keys, and each model's entry in the export's provenance records its section, registry, entry id and entry digest.
If some entries were trained on another genome than the run's [variants] reference, the run warns once for the
section, with how many and on which genomes, and scores them on the run's reference.
Each model's export columns are prefixed with a short name made from its key and 12 hex digits of the key's hash:
lab_encsr685zmp__<hash>_<column>. The name keeps the key's first 64 characters, so a long key still gives a short
prefix, and the hash keeps two keys that share those characters apart. The store and the provenance keep the whole
key. The export has one row per variant, with a block of columns for each model, so a list of hundreds of models
gives a wide file. Its _columns.tsv names the model key and the score of each column
(Find a model's columns). [export] layout = "long" writes the scores as a
table with a row per variant and model instead, one table for all the models of a binding
(Long format for many models).
The genomes¶
Each genome's id is <assembly>/<authority>/<product>: GRC's name for the assembly, who publishes the file, and
the publisher's own name for it. Names ignore case. UCSC's genomes also answer to UCSC's names.
| Id | Alias | What it is |
|---|---|---|
grch38/ucsc/hg38 |
hg38 |
Human GRCh38 as UCSC publishes it, hg38.fa: the initial release, with its 261 alt contigs, repeats in lowercase. The quickstart uses it. |
grch38/ncbi/no-alt-analysis-set |
NCBI's GRCh38 no-alt analysis set, GCA_000001405.15_GRCh38_no_alt_analysis_set.fna, byte for byte the FASTA ENCODE distributes: no alt contigs and no decoys, plus chrEBV, all uppercase. |
|
grch37/ucsc/hg19 |
hg19 |
Human GRCh37 as UCSC publishes it, hg19.fa: the initial release, with its 9 alternate haplotypes. Its chrM is NC_001807, not the rCRS. |
grch37/encode/male-hg19 |
Human GRCh37 as ENCODE distributes it, male.hg19: hg19's 25 chromosomes only. |
|
grcm39/ucsc/mm39 |
mm39 |
Mouse GRCm39 as UCSC publishes it, mm39.fa. |
grcm38/ucsc/mm10 |
mm10 |
Mouse GRCm38 as UCSC publishes it, mm10.fa. |
grcm38/encode/mm10-no-alt-analysis-set |
Mouse GRCm38 as ENCODE distributes it, mm10_no_alt_analysis_set_ENCODE: mm10's 66 sequences. |
hg38 and NCBI's analysis set are both GRCh38, but their sequences differ on 18 of the 25 chromosomes: UCSC
writes N where the assembly has another ambiguity code, which the analysis set keeps, and the analysis set writes N
over chrY's pseudoautosomal regions and over duplicate centromeric arrays on chr5, 14, 19, 21 and 22. ENCODE's two
references differ from UCSC's in sequence on chrY only, whose pseudoautosomal regions they write as N. Some models were trained or validated on one and some on another. Use
the one your other analyses use.
Add a registry¶
A registry someone else publishes is an address: a web page, a bucket or a Hugging Face repository. Add it to a
run spec under [registries], with the name you will type:
Then pass the spec to altar resources, so it knows the registry and uses the spec's cache:
The first command reads the registry's index and keeps it in the cache; --refresh reads it again later, to see
entries added since. A run reads only the cache. To have a registry in every spec and command, list it in
ALTAR_REGISTRIES as name=address, separated by commas:
A registry in a public bucket or repository needs nothing more: without a connection it is read anonymously, never with your own cloud credentials. A registry in a private bucket is read through a connection, which reads the environment's credentials:
[registries]
lab = { uri = "gs://lab-models/registry.json", connection = "gcs" }
[connections.gcs]
type = "gcp"
A registry is someone else's document, so a fetch limits what one that is not on your disk can make it read:
- Only
https://,gs://,s3://andhf://addresses. A plainhttp://registry or file is refused, and so is a redirect fromhttps://to anything else. - No file on your machine, and no host on your own network: a fetch refuses an address that is written as an IP address, or whose name points at this machine, a private network or a cloud metadata service.
- Cloud files are read anonymously unless the registry reads through a connection, so a registry cannot make a fetch use your own credentials.
- A download stops once it is larger than its entry says. When the entry gives no size, the limit is 64 GB; set
ALTAR_FETCH_MAX_BYTESto a number of bytes to change it.
A registry on your intranet needs your opt-in, on its table or on the connection it reads through:
[registries]
mirror = { uri = "https://models.lab.internal/registry.json", allow_private_hosts = true }
A registry on your own disk is yours, and may name any file.
Publish a registry shows how to publish your own models.
Use your own files¶
Paths still work. Use them for a FASTA or weights no registry lists:
[variants]
path = "cohort.tsv"
reference_fasta = "my-genomes/GRCh38.fa"
[models.legnet-k562]
type = "LEGNET"
resource_manifest = "legnet-k562/legnet.resources.json"
Set reference or reference_fasta, not both; and resources or resource_manifest, not both. With your own
FASTA, reference_digest checks it against a digest you give; see
Pin the reference by its digest. A model's page says how to package
other weights, such as LegNet's.
Why a named entry matters¶
A model that reads the genome identifies it by the FASTA's digest, and identifies its weights by theirs. Those
digests are part of its scores' identity. Two runs reuse each other's stored scores only when the digests match,
wherever the files were read from. Everyone who fetches hg38 and legnet/wtc11 gets the same bytes, so their
scores are reused across runs, machines and people. A FASTA with other bytes, even of the same assembly, scores
under its own identity.
Score reuse and provenance explains how stored scores are matched.