Skip to content

Publish a registry

A registry is an index of models and reference genomes that anyone can host. Each item in it is an entry: a model's saved configuration and the files it names, or a genome's FASTA. A user adds your registry to a run spec, finds entries with altar resources, fetches one with altar resources fetch <registry>/<id>, and names it in the spec as resources = "<registry>/<id>". Altar ships two registries this way: genome, with the reference genomes, and legnet, with LegNet's published checkpoint.

This page is the format's specification: the two documents, the rules a fetch checks, and how to host and check a registry. The format, altar resources and the altar.registries entry-point group are Provisional.

What a registry is

A registry is a set of static files:

registry.json                     the index: one line per entry; the only file that changes
entries/sha256/<hex>.json         one entry document each, named by the SHA-256 of its bytes

The files an entry names can live anywhere: beside the index, in a bucket, or at the publisher who first released them, inside an archive. Each one is named by its SHA-256 digest, so Altar checks every byte it fetches and identifies every file by its digest, never by its address.

An entry document never changes once published. To change an entry, publish a new document, whose path has a new digest, and point the index line at it. Keep the old document if anyone may have pinned it.

The index

The index, altar-registry/1, names the registry and lists its entries. A listing and its filters read only the index, so each line holds what a listing shows and nothing more.

Key Required What it is
format yes "altar-registry/1"
name yes The registry's name. A user can add it under another name.
description no One sentence for altar resources.
entries yes The index lines, below.

Each line of entries:

Key Required What it is
id yes The entry's id, unique in the registry ignoring case, at most 128 characters: one or more segments of letters, digits, ., _ and -, joined by single /. No segment is empty, . or .., and the id neither starts nor ends with /. Users type it after the registry's name, such as lab/ENCSR296UHQ or lab/k562/dnase: the registry is the name's first segment, and the id the rest. A model listed under it is keyed <registry>-<id> in lowercase, each / written - and each . written _, such as lab-k562-dnase; two ids that would share a key cannot be listed in one run.
aliases no For a genome, other names that name it as its id does, such as hg38; unique with every id and alias of the registry, ignoring case. A model entry has none.
name yes The entry's display name, which a listing leads with.
type yes The model type a binding registers, such as CHROMBPNET, or "GENOME".
plugin_id for a model The binding's plugin id, such as org.kundajelab.altar.model.chrombpnet.
training_reference no {genome_build, digest}: the genome a model was trained on, and its FASTA's digest. For a genome, the genome itself.
size_bytes no What the entry's files take once fetched.
facets no The metadata a listing filters on (see Metadata).
entry one of entry and document The entry document's path, relative to the index, ending in sha256/<hex>.json.
document one of entry and document The entry document itself, inline, for a small registry shipped in a package.
page no A web page about the entry: an https:// URL without a credential.
{
  "format": "altar-registry/1",
  "name": "lab",
  "description": "The lab's ChromBPNet models",
  "entries": [
    {
      "id": "ENCSR296UHQ",
      "name": "ChromBPNet, K562 DNase-seq",
      "type": "CHROMBPNET",
      "plugin_id": "org.kundajelab.altar.model.chrombpnet",
      "training_reference": {
        "genome_build": "hg38",
        "digest": "sha256:9cce8b926416dd96b152deea85188495b75f7ac8d634cc723a017067be8702b7"
      },
      "facets": {
        "assay": "DNase-seq",
        "cell_types": ["K562"],
        "organism": "Homo sapiens",
        "project": "ENCODE"
      },
      "entry": "entries/sha256/b3ce2b2c41d3a947b5bfc481e946b6afde98cadc955c4bdfe410b0484f2e3697.json",
      "page": "https://www.encodeproject.org/ENCSR296UHQ/"
    }
  ]
}

This index lists ENCODE's K562 DNase-seq ChromBPNet model, trained on NCBI's GRCh38 no-alt analysis set, genome/grch38/ncbi/no-alt-analysis-set, whose FASTA digest training_reference gives. The hexadecimal digits in entry are the SHA-256 of the entry document's bytes: here, of the example in A model's configuration, saved exactly as shown with a final newline. Altar checks the document it fetches against them, so the index pins every entry it lists. An inline document's digest is taken over its canonical JSON: keys sorted, no spaces, UTF-8.

A relative URI, in the index or in any entry, is relative to the index: entries/sha256/<hex>.json and files/weights.h5 are beside registry.json, however deep the document that names them.

An entry

An entry document, altar-registry-entry/1, holds everything a run needs: a model's configuration, or a genome's FASTA, with every file named by its digest.

Key Required What it is
format yes "altar-registry-entry/1"
id, type yes The same as the index line's.
name, description no As a listing shows them.
configuration for a model The model's saved configuration (below).
prepared no Files the model's preparation step writes, so a run need not run altar prepare (see Prepared files).
reference with prepared The genome the model was trained on and its prepared files made from: {genome_build, genome: {digest}, genome_index: {digest}}, the digests of its FASTA and of the FASTA's .fai.
genome for a genome The genome's FASTA and contigs (see A genome entry).
license no The terms the files are published under, or where they are stated.
citation no How to cite the model or genome.
page no A web page about the entry, such as its page in a catalog: an https:// URL without a credential. altar resources REGISTRY/ID shows it, or the index line's page when the entry gives none.
metadata no What the entry describes (see Metadata).
upstream no [{url, description, source_checksum}]: where the files first came from, recorded for provenance. Altar never fetches it.

A model's configuration

configuration is the saved configuration a binding writes with save_configuration, and reads back with load_configuration:

Key What it is
format "altar.model-configuration"
plugin_id, schema_id, schema_version The binding and the version of its configuration schema. A newer binding migrates an older version.
plugin_version Optional: the binding's version that wrote it.
configuration The binding's configuration fields.

Every object in configuration.configuration that has a uri is a file reference: Altar fetches it, and before it hands the configuration to the binding it replaces the uri with the path of the fetched copy and removes the keys only a fetch reads (member, source_checksum, size_bytes, compression). The binding and its runtime see local files. Its identity reads each file's digest, never its uri, so the scores are the same, and are reused, whether a run read the files from a registry or from anywhere else.

This is the entry of ENCODE's K562 DNase-seq ChromBPNet model. It takes its five folds and its peaks from the two archives ENCODE publishes, as archive members, each with the member's digest and the archive's:

{
  "format": "altar-registry-entry/1",
  "id": "ENCSR296UHQ",
  "type": "CHROMBPNET",
  "name": "ChromBPNet, K562 DNase-seq",
  "configuration": {
    "format": "altar.model-configuration",
    "plugin_id": "org.kundajelab.altar.model.chrombpnet",
    "schema_id": "org.kundajelab.altar.model.chrombpnet.configuration",
    "schema_version": "0.0.3",
    "configuration": {
      "weights": [
        {
          "uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
          "member": "fold_0/model.chrombpnet_nobias.fold_0.ENCSR000EOT.h5",
          "digest": "sha256:920780b6855e45efa9c62bb00b79f60ed21b64c371fb01145cf7f83468cb8bfd",
          "source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
        },
        {
          "uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
          "member": "fold_1/model.chrombpnet_nobias.fold_1.ENCSR000EOT.h5",
          "digest": "sha256:26fa44f7f356f89aa55bfe24e71c9d84ca10187cd211b6db57a732d2146d4d7e",
          "source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
        },
        {
          "uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
          "member": "fold_2/model.chrombpnet_nobias.fold_2.ENCSR000EOT.h5",
          "digest": "sha256:85f352becdadcbffd640fd06e1dd0fc5025b15dc704db105b151ebe981c5643d",
          "source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
        },
        {
          "uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
          "member": "fold_3/model.chrombpnet_nobias.fold_3.ENCSR000EOT.h5",
          "digest": "sha256:95a70fbca9197f7ded8bba430d3d47db5484a4b68240fc8d854efafecc210454",
          "source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
        },
        {
          "uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
          "member": "fold_4/model.chrombpnet_nobias.fold_4.ENCSR000EOT.h5",
          "digest": "sha256:e05929e0443c9fb9955b395ee09e5b1995cf696ca7702e074d4d242a9e11db6f",
          "source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
        }
      ],
      "peaks": {
        "uri": "https://www.encodeproject.org/files/ENCFF926SHM/@@download/ENCFF926SHM.tar.gz",
        "member": "peaks.all_input_regions.ENCSR000EOT.bed.gz",
        "digest": "sha256:830c201f4ff43b3080e1b319a89f78a52f9fea7b931824ce2a05a2ee5243db0a",
        "source_checksum": {"sha256": "ea1a1061c887988b1d29b8ad3fad7b39899a8457f010258a22f395ad39784546"}
      },
      "training_genome_build": "hg38",
      "image": "riyavsinha/altar-chrombpnet@sha256:935bdc0e20bbeaa04c10bfeb8a705a6aecea49b66742b81d47ef2dcfe120d4ee"
    }
  },
  "reference": {
    "genome_build": "hg38",
    "genome": {"digest": "sha256:9cce8b926416dd96b152deea85188495b75f7ac8d634cc723a017067be8702b7"},
    "genome_index": {"digest": "sha256:502a1b8fb73ccd53285c28a0f12df90c818b4fe3de1e862ef47c593ef1a0a4b4"}
  },
  "license": "ENCODE data, free to use; https://www.encodeproject.org/help/citing-encode/ says how to cite them",
  "citation": "Pampari A et al. ChromBPNet: bias factorized, base-resolution deep learning models of chromatin accessibility reveal cis-regulatory sequence syntax, transcription factor footprints and regulatory variants. bioRxiv (2024). doi:10.1101/2024.12.25.630221",
  "metadata": {
    "assay": "DNase-seq",
    "cell_types": ["K562"],
    "organism": "Homo sapiens",
    "project": "ENCODE"
  },
  "upstream": [
    {
      "url": "https://www.encodeproject.org/ENCSR296UHQ/",
      "description": "The ENCODE model dataset: ENCFF574YLK holds the five folds, ENCFF926SHM the peaks"
    },
    {
      "url": "https://www.encodeproject.org/experiments/ENCSR000EOT/",
      "description": "The ENCODE DNase-seq experiment the model was trained on"
    }
  ]
}

A genome entry

A genome entry has type = "GENOME" and holds genome in place of configuration:

Key Required What it is
genome_build yes The assembly, such as hg38 or mm10. A run's [run] genome_build must name the same one.
fasta yes The FASTA, a file reference.
fai no {digest, uri}: the FASTA's .fai index. Without uri, Altar writes the index and checks it against digest.
contigs no Every contig of the FASTA, in order: {name, canonical, length, digest}, with the name a variant key gives it and its GA4GH refget digest. A fetch checks the index against them.
collections no {seqcol, canonical}: GA4GH sequence-collection level-0 digests of the contigs, and of their canonical names.
source no Who publishes the FASTA: {authority, accession, product, file_name, url, description}. authority is NCBI, UCSC or ENCODE; accession the assembly accession, such as GCA_000001405.15; product the publisher's own name for the file; url its https:// page; and description one line on what sets this file apart from others of its assembly. authority, product and description are required.

Altar's own genomes are named <assembly>/<authority>/<product>, such as grch38/ucsc/hg38: GRC's name for the assembly, who publishes the file, and the publisher's own name for it.

{
  "format": "altar-registry-entry/1",
  "id": "grch38/ucsc/hg38",
  "type": "GENOME",
  "name": "Human GRCh38, UCSC hg38.fa",
  "genome": {
    "genome_build": "hg38",
    "source": {
      "authority": "UCSC",
      "accession": "GCA_000001405.15",
      "product": "hg38",
      "file_name": "hg38.fa.gz",
      "url": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/",
      "description": "Initial GRCh38 release with its 261 alt contigs and no fix patches; soft-masked (repeats in lowercase); ambiguity codes written as N"
    },
    "fasta": {
      "uri": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz",
      "compression": "gzip",
      "digest": "sha256:5be01555d98347fdb3714dc84c6f77c9d8bc774adcf32c6f7a8fa06f5baf5e51",
      "size_bytes": 3273481150,
      "source_checksum": {"sha256": "c1dd87068c254eb53d944f71e51d1311964fce8de24d6fc0effc9c61c01527d4"}
    },
    "fai": {"digest": "sha256:3b425de206296a5c8053023fa5ca61da43cfe78c1737c12e58c83367c7e83c21"},
    "contigs": [
      {"name": "chr1", "canonical": "chr1", "length": 248956422, "digest": "SQ.2YnepKM7OkBoOrKmvHbGqguVfF9amCST"}
    ]
  }
}

A run that names a genome, [variants] reference = "hg38", looks it up in the genome registry, by id or by alias. A model entry's reference names its genome by digest, so Altar can say which genome entry it is.

File references

Every file an entry names, in its configuration, its prepared files or its genome, takes one of two forms.

A direct copy, {uri, digest}: the bytes at uri are the file.

An archive member, {uri, member, digest, source_checksum}: uri is a tar, tar.gz or zip archive, and member is the path of the file inside it. Use it to point at the archive a publisher first released, such as ENCODE's model tarballs, instead of hosting a copy.

Key Required What it is
uri yes https://, gs://, s3://, hf://, or a path relative to the index, such as files/weights.h5, in every document.
digest yes sha256: and 64 lowercase hexadecimal digits: the digest of the file Altar keeps, the member itself for an archive member, unpacked for a compressed copy.
member for an archive member The member's path inside the archive.
source_checksum no The publisher's checksum of the bytes at uri, such as {"md5": "…"} or {"sha256": "…"} (plain hexadecimal; md5, sha1, sha256 or sha512). Checked as the bytes arrive.
compression no "gzip": the bytes at uri are a gzip stream of the file, unpacked as they arrive. Not with member.
size_bytes no The size of the file Altar keeps. A fetch stops a download that grows past it; give it for every large file.
revision no A provider revision, a string or null, as an Altar saved configuration's file references hold it. Identity reads only digest.

A fetch downloads each archive once, however many members an entry takes from it, extracts the members it needs and checks each against its digest. It deletes the archive afterwards unless --keep-archives is given. A file whose bytes do not match its digest, because the publisher replaced it at the same address, stops the fetch with an error that names the entry, the URL and both digests. Altar never accepts it.

A download also stops once it is larger than the file's size_bytes, or, for a file without one and for an archive, than 64 GB, which the reader's ALTAR_FETCH_MAX_BYTES changes. Give size_bytes for every large file. A gzip file is unpacked a megabyte at a time and stops at the same limit, so a small download cannot unpack into a disk-filling one. In an archive, a member stored as a hard link to an earlier member is read from that member; a symbolic link is refused. Members are read into the store by digest and never written under their names, so no name in an archive reaches the reader's files.

digest names the file in both forms, so you can move a file from an archive member to a direct copy, or between hosts, without changing any identity: runs that read the old address reuse scores computed from the new one.

Every file needs its digest before it is published. An archive member whose digest you do not know yet cannot be listed: extract it once and hash it.

Metadata

metadata describes what an entry models. Every key is optional, and you may add others:

Key Value
assay one value
cell_types, organs, systems, life_stages a list of values
sex, biosample, treatment, organism, source one value
project one value, such as "ENCODE"

A value is an ontology term, {"id": "CL:0000084", "name": "T cell"}, or plain text when no term applies.

An index line's facets repeat the values of assay, cell_types, organs, systems, life_stages, sex, organism and project, so a listing filters without reading every entry. Give only those keys, with the same values as the entry's metadata.

altar resources <registry> --where cell_types=k562 matches a facet ignoring case, and a term by its id or its name. Repeated --where options must all match. --search dnase k562 matches the words of an entry's name and of its facets' names, each word as a prefix.

Prepared files

Some models read files a preparation step writes, such as ChromBPNet's calibration files, which altar prepare publishes under models/<model key>/ in the cache. An entry that lists them in prepared saves every user that step: a run puts them where altar prepare would have, and scores without it.

Key What it is
path Where altar prepare publishes the file, relative to models/<model key>/.
resource The file, a file reference.
"prepared": [
  {
    "path": "preprocessing.manifest.json",
    "resource": {
      "uri": "https://example.org/lab/my-model/preprocessing.manifest.json",
      "digest": "sha256:…"
    }
  }
]

The binding checks prepared files as it checks the ones altar prepare publishes, so they must be the files preparation would have written for this configuration and reference.

Digests and pinning

A digest is the SHA-256 of a file's bytes. Altar uses three kinds:

  • A file's digest identifies the file. It enters the identity of the model's scores, so two runs that fetch the same files reuse each other's scores, wherever the files were hosted.
  • An entry's digest, the hexadecimal digits in its path, identifies the entry document. A user who writes resources = "lab/ENCSR296UHQ@sha256:<hex>" gets exactly that document, even after the index points elsewhere, as long as it is still published or already fetched.
  • source_checksum is the publisher's checksum of the bytes at a URI, and only checks a download.

A run records the registry's name and address, the entry's id and the entry's digest in its provenance. None of them enters identity.

Rules a check enforces

The JSON Schemas below state each document's shape. altar resources check also checks what a schema cannot:

  • Each entry path ends in sha256/<hex>.json, and the document there has that digest.
  • Each document parses, and its id and type are its index line's. Ids and aliases are unique ignoring case, and only a genome has aliases.
  • A model entry's plugin_id is its configuration's. When the binding is installed, the configuration loads.
  • training_reference agrees with the entry's reference, and facets with its metadata.
  • Every file has a digest. Every URI is https://, gs://, s3://, hf://, or relative; never http://, a local path, or a URL with a credential (a user name or password, or a signed-URL or token query parameter). Users pass credentials through a connection.

A fetch holds a registry hosted elsewhere to the same rules, its own address included, after it resolves each relative URI against the index, and refuses anything else before reading it. It also connects only to public hosts. A host written as an IP address is refused, and so is a host whose name points at the reader's machine, a private network or a cloud metadata service; the address is checked when the fetch connects, so a name cannot point somewhere else between the check and the download. A redirect is checked as a new address, may not leave https://, and a fetch follows at most five. A fetch uses no HTTP proxy. Only a registry on the reader's own disk may name local files or http://.

A gs://, s3:// or hf:// file is read anonymously unless the reader names a connection for the registry, so publish files that anyone can read. A registry on an intranet, whose hosts are on a private network, needs its readers' opt-in: allow_private_hosts = true on its [registries] table or on the connection it reads through.

Check a registry

Run the check on the index before you publish it, and in your publishing CI:

altar resources check https://example.org/registry/registry.json

It reads the index and every entry it names, but downloads no other file. It prints each problem on its own line and exits with status 2 if there is one, or 0 if there is none. --json prints the problems as JSON. In Python, altar.testing.check_registry(uri) returns the same list of problems.

Host it

Put registry.json and entries/ in one directory, and publish it anywhere Altar reads:

Where A user adds it as
A web server, such as GitHub Pages lab = "https://example.org/registry/registry.json"
A Google Cloud Storage bucket lab = "gs://lab-models/registry.json"
A Hugging Face repository lab = "hf://datasets/lab/models/registry.json"
A directory on a shared disk lab = "/shared/registries/lab/registry.json"

under [registries] in a run spec. Without a connection, a gs://, s3:// or hf:// registry and the files it names are read anonymously, never with the reader's own cloud credentials, so a public bucket or repository works as it is. A private one is read with a connection: lab = { uri = "gs://lab-models/registry.json", connection = "lab" }, and the run spec's [connections.lab] gives its options. Never put a credential in the registry itself.

A registry on an intranet, whose hosts are on a private network, needs the reader's opt-in: lab = { uri = "https://models.lab.internal/registry.json", allow_private_hosts = true }.

Ship a registry in a package

A binding can ship a registry with its package. Put the index, with entries inline or beside it, in the package, and register a PackagedRegistry subclass under the altar.registries entry-point group:

[project.entry-points."altar.registries"]
lab = "lab_models.registry:LabRegistry"
from pathlib import Path

from altar.registries import PackagedRegistry


class LabRegistry(PackagedRegistry):
    name = "lab"

    @classmethod
    def index_path(cls) -> Path:
        return Path(__file__).with_name("registry.json")

The entry point's name is the registry's name. Once the package is installed, altar resources lists it.

JSON Schemas