Skip to content

Find and fetch models and genomes

A run names its models and its reference genome by entries of a registry. A registry is an index of models and genomes that anyone can host, and each entry names its files at fixed addresses, each with a recorded SHA-256 digest. altar resources finds entries, altar resources fetch downloads them and checks every file, and a run spec then names the entry instead of giving paths.

altar resources, the registry format and the run-spec keys on this page are Provisional.

See the registries

altar resources

The listing has one row per registry: its name, how many entries it lists, where it comes from, and what it holds.

  NAME    ENTRIES  FROM          DESCRIPTION
  genome  7        altar         Reference genome FASTAs, named in a run spec as [variants] reference
  legnet  1        altar-legnet  Published LegNet checkpoints, which predict MPRA regulatory activity

Altar itself ships genome, with the reference genomes. Each model package you install can add its own; this legnet registry comes from altar-legnet. Add a registry shows how to add one someone else publishes.

Find an entry

Name a registry to list its entries:

altar resources genome --where "organism=mus musculus"
  ID                                      ALIAS  BUILD  AUTHORITY  ACCESSION        PRODUCT                          SIZE    FETCHED
  grcm39/ucsc/mm39                        mm39   mm39   UCSC       GCA_000001635.9  mm39                             2.8 GB  no
  grcm38/ucsc/mm10                        mm10   mm10   UCSC       GCA_000001635.2  mm10                             2.8 GB  no
  grcm38/encode/mm10-no-alt-analysis-set  -      mm10   ENCODE     -                mm10_no_alt_analysis_set_ENCODE  2.8 GB  no

  grcm39/ucsc/mm39
    61 sequences: the chromosomes, random and chrUn contigs, no alt or fix contigs; soft-masked

  grcm38/ucsc/mm10
    Initial GRCm38 release, 66 sequences with no alt or fix patches; soft-masked

  grcm38/encode/mm10-no-alt-analysis-set
    mm10.fa's 66 sequences, with chrY's pseudoautosomal region hard-masked with N; soft-masked

A genome's row leads with its id, <assembly>/<authority>/<product>: GRC's name for the assembly, who publishes the file, and the publisher's own name for it. Then come its alias, if it has one, and who publishes it, followed by a line on what sets each file apart. A model's row leads with the entry's name, then its id. You copy the id into commands and specs as <registry>/<id>, such as genome/grcm38/ucsc/mm10 or legnet/wtc11; an id may hold / itself. BUILD is the genome the entry is on: a genome's own, or the one a model was trained on.

  • --where KEY=VALUE keeps the entries whose KEY is VALUE, ignoring case. The keys are name, id, type, genome_build, and what a registry describes its entries by: assay, cell_types, organs, systems, life_stages, sex, organism and project. An ontology term matches its id or its name, so --where cell_types=K562 and --where cell_types=EFO:0002067 find the same entries. Repeat --where to require each one.
  • --search WORDS keeps the entries with a word of their name, id or description terms that starts with each word you give: --search dnase k562.

Name one entry to see everything its registry says about it: its files with their digests and addresses, the genome it was trained on, its metadata, how to cite it, its terms, and whether you have fetched it.

altar resources legnet/wtc11

Fetch one

altar resources fetch hg38 legnet/wtc11

A genome needs no genome/: its id, such as grch38/ucsc/hg38, or its alias, such as hg38, names it, and so does any name whose first part is not a registry's. The command downloads each file the entry names, unpacks it if the publisher compressed it, and checks its digest. It prints a line per file and ends with the run-spec lines that name the entry. A genome takes a few minutes, and also gets an index, checked against its contigs.

Some entries take files from an archive a publisher released, such as ENCODE's model tarballs. The fetch downloads each archive once, extracts the files the entry needs and checks each one, then deletes the archive. Pass --keep-archives to keep it.

The files go into resources/sha256/ in the cache, .altar/ in the working directory, the same cache a run keeps its scores in. Each file is kept under its digest, so one that two entries name is stored once. To keep them somewhere else, set ALTAR_CACHE_DIR, or pass --cache-dir to the command:

export ALTAR_CACHE_DIR=/data/altar-cache
altar resources fetch hg38

A run looks for fetched files in its cache: [run] cache_dir, or ALTAR_CACHE_DIR, or ./.altar. A spec that sets cache_dir needs the files fetched there too, with --cache-dir and the same directory.

Each file gets a verification record beside it. Running the command again skips every file whose record still matches it, so an interrupted fetch resumes. A file whose digest does not match, because its publisher replaced it at the same address, is deleted, and the command stops with exit status 2 and names the entry, the address and both digests. A run never downloads anything.

Name it in a run spec

A fetched genome goes in [variants] reference, and a fetched model in [models.<name>] resources:

[variants]
path = "cohort.tsv"
reference = "hg38"

[models.legnet-wtc11]
resources = "legnet/wtc11"

The entry gives the model's type, LEGNET, and its whole configuration. A field under [models.legnet-wtc11.configuration], such as average_reverse = false, changes the entry's value. The run reads the files from the cache, and takes the genome's digest from its entry, so it does not hash 3 GB on every run. The genome must be on [run] genome_build.

resources = "legnet/wtc11@sha256:<hex>" pins the entry: the run reads exactly that entry document, even after the registry lists a newer one. altar resources legnet/wtc11 prints the entry's digest.

If an entry has not been fetched, altar run and altar run --validate stop with exit status 2 and print the command to run:

altar run: error: variants.reference: genome/grch38/ucsc/hg38 has not been fetched into .altar; run `altar resources fetch genome/hg38` first

A run records each model's registry, the registry's address, the entry's id and the entry's digest in its summary and its export's provenance.

Score with many models

To score with a whole compendium, such as hundreds of ChromBPNet models, list the entries in one section:

[models.compendium]
resources = [
  "lab/ENCSR685ZMP",
  "lab/ENCSR296UHQ",
  "lab/ENCSR000EOT",
]
max_variants_per_batch = 50000

The section becomes one model per entry, in the list's order. Every other key of the section, such as configuration, output, backend, pool or max_variants_per_batch, applies to each of them. Entries may come from different registries. A type must be every entry's type, and model_name is refused, since each model is named by its key.

Each model's key is the entry's registry and id joined by -, in lowercase, with each / of the id written as - and each . as _: lab/ENCSR685ZMP is the model lab-encsr685zmp, lab/v1.2 is lab-v1_2, and lab/k562/dnase is lab-k562-dnase. Two entries that would get one key, such as lab/a/b and lab/a-b, cannot be listed in one run; name one in a section of its own. The key never comes from the entry's display name, so a registry can rename an entry without changing it. It does not depend on the section either: the list above scores the same models, and reuses the same stored scores, as one section per entry would:

[models.lab-encsr685zmp]
resources = "lab/ENCSR685ZMP"

So you can move an entry from one section to another, or from its own section into a list, and keep its scores. A spec that names one entry twice, or two entries whose keys are the same, such as lab/v1.2 and lab/v1_2, is refused, and so is a list entry whose key is another section's.

Fetch every entry a spec names, its [variants] reference too, with one command:

altar resources fetch --spec score.toml

Before it downloads anything, it says how much there is. For a spec that lists 1,500 models and names a genome:

score.toml names 1,501 entries: 1,200 already fetched, 301 to fetch (45.2 GB) into .altar

A fetch larger than 20 GB asks you to confirm it first, and so does one with an entry whose registry does not give its size; --yes confirms it, and without a terminal the command stops and says the size, or how many entries have none. Set ALTAR_FETCH_CONFIRM_BYTES to a number of bytes to change the threshold. --jobs N fetches up to N entries at once, 4 by default.

altar run --validate checks the list without downloading anything. When some entries are missing, it says how many, the size still to fetch and the command to run:

altar run: error: models.compendium.resources: 301 of 1,500 entries have not been fetched into .altar (45.2 GB to fetch), such as lab/ENCSR000EOT; run `altar resources fetch --spec score.toml` first, which fetches every entry the spec names

When every entry is fetched, the check prints one line for the section, with its count, in place of a line per model. A run prints a line before the section's first model with how many it holds, one line per model as it scores, with its place in the list, and a line after the last model with the section's totals. The run summary lists the section with its count and its models' keys, and each model's entry in the export's provenance records its section, registry, entry id and entry digest.

If some entries were trained on another genome than the run's [variants] reference, the run warns once for the section, with how many and on which genomes, and scores them on the run's reference.

Each model's export columns are prefixed with a short name made from its key and 12 hex digits of the key's hash: lab_encsr685zmp__<hash>_<column>. The name keeps the key's first 64 characters, so a long key still gives a short prefix, and the hash keeps two keys that share those characters apart. The store and the provenance keep the whole key. The export has one row per variant, with a block of columns for each model, so a list of hundreds of models gives a wide file. Its _columns.tsv names the model key and the score of each column (Find a model's columns). [export] layout = "long" writes the scores as a table with a row per variant and model instead, one table for all the models of a binding (Long format for many models).

The genomes

Each genome's id is <assembly>/<authority>/<product>: GRC's name for the assembly, who publishes the file, and the publisher's own name for it. Names ignore case. UCSC's genomes also answer to UCSC's names.

Id Alias What it is
grch38/ucsc/hg38 hg38 Human GRCh38 as UCSC publishes it, hg38.fa: the initial release, with its 261 alt contigs, repeats in lowercase. The quickstart uses it.
grch38/ncbi/no-alt-analysis-set NCBI's GRCh38 no-alt analysis set, GCA_000001405.15_GRCh38_no_alt_analysis_set.fna, byte for byte the FASTA ENCODE distributes: no alt contigs and no decoys, plus chrEBV, all uppercase.
grch37/ucsc/hg19 hg19 Human GRCh37 as UCSC publishes it, hg19.fa: the initial release, with its 9 alternate haplotypes. Its chrM is NC_001807, not the rCRS.
grch37/encode/male-hg19 Human GRCh37 as ENCODE distributes it, male.hg19: hg19's 25 chromosomes only.
grcm39/ucsc/mm39 mm39 Mouse GRCm39 as UCSC publishes it, mm39.fa.
grcm38/ucsc/mm10 mm10 Mouse GRCm38 as UCSC publishes it, mm10.fa.
grcm38/encode/mm10-no-alt-analysis-set Mouse GRCm38 as ENCODE distributes it, mm10_no_alt_analysis_set_ENCODE: mm10's 66 sequences.

hg38 and NCBI's analysis set are both GRCh38, but their sequences differ on 18 of the 25 chromosomes: UCSC writes N where the assembly has another ambiguity code, which the analysis set keeps, and the analysis set writes N over chrY's pseudoautosomal regions and over duplicate centromeric arrays on chr5, 14, 19, 21 and 22. ENCODE's two references differ from UCSC's in sequence on chrY only, whose pseudoautosomal regions they write as N. Some models were trained or validated on one and some on another. Use the one your other analyses use.

Add a registry

A registry someone else publishes is an address: a web page, a bucket or a Hugging Face repository. Add it to a run spec under [registries], with the name you will type:

[registries]
lab = "https://example.org/registry/registry.json"

Then pass the spec to altar resources, so it knows the registry and uses the spec's cache:

altar resources lab --spec score.toml
altar resources fetch lab/ENCSR296UHQ --spec score.toml

The first command reads the registry's index and keeps it in the cache; --refresh reads it again later, to see entries added since. A run reads only the cache. To have a registry in every spec and command, list it in ALTAR_REGISTRIES as name=address, separated by commas:

export ALTAR_REGISTRIES="lab=https://example.org/registry/registry.json"

A registry in a public bucket or repository needs nothing more: without a connection it is read anonymously, never with your own cloud credentials. A registry in a private bucket is read through a connection, which reads the environment's credentials:

[registries]
lab = { uri = "gs://lab-models/registry.json", connection = "gcs" }

[connections.gcs]
type = "gcp"

A registry is someone else's document, so a fetch limits what one that is not on your disk can make it read:

  • Only https://, gs://, s3:// and hf:// addresses. A plain http:// registry or file is refused, and so is a redirect from https:// to anything else.
  • No file on your machine, and no host on your own network: a fetch refuses an address that is written as an IP address, or whose name points at this machine, a private network or a cloud metadata service.
  • Cloud files are read anonymously unless the registry reads through a connection, so a registry cannot make a fetch use your own credentials.
  • A download stops once it is larger than its entry says. When the entry gives no size, the limit is 64 GB; set ALTAR_FETCH_MAX_BYTES to a number of bytes to change it.

A registry on your intranet needs your opt-in, on its table or on the connection it reads through:

[registries]
mirror = { uri = "https://models.lab.internal/registry.json", allow_private_hosts = true }

A registry on your own disk is yours, and may name any file.

Publish a registry shows how to publish your own models.

Use your own files

Paths still work. Use them for a FASTA or weights no registry lists:

[variants]
path = "cohort.tsv"
reference_fasta = "my-genomes/GRCh38.fa"

[models.legnet-k562]
type = "LEGNET"
resource_manifest = "legnet-k562/legnet.resources.json"

Set reference or reference_fasta, not both; and resources or resource_manifest, not both. With your own FASTA, reference_digest checks it against a digest you give; see Pin the reference by its digest. A model's page says how to package other weights, such as LegNet's.

Why a named entry matters

A model that reads the genome identifies it by the FASTA's digest, and identifies its weights by theirs. Those digests are part of its scores' identity. Two runs reuse each other's stored scores only when the digests match, wherever the files were read from. Everyone who fetches hg38 and legnet/wtc11 gets the same bytes, so their scores are reused across runs, machines and people. A FASTA with other bytes, even of the same assembly, scores under its own identity.

Score reuse and provenance explains how stored scores are matched.