Publish a registry¶
A registry is an index of models and reference genomes that anyone can host. Each item in it is an entry: a
model's saved configuration and the files it names, or a genome's FASTA. A user adds your registry to a run spec,
finds entries with altar resources, fetches one with altar resources fetch <registry>/<id>, and names it in
the spec as resources = "<registry>/<id>". Altar ships two registries this way: genome, with the reference
genomes, and legnet, with LegNet's published checkpoint.
This page is the format's specification: the two documents, the rules a fetch checks, and how to host and check
a registry. The format, altar resources and the altar.registries entry-point group are
Provisional.
What a registry is¶
A registry is a set of static files:
registry.json the index: one line per entry; the only file that changes
entries/sha256/<hex>.json one entry document each, named by the SHA-256 of its bytes
The files an entry names can live anywhere: beside the index, in a bucket, or at the publisher who first released them, inside an archive. Each one is named by its SHA-256 digest, so Altar checks every byte it fetches and identifies every file by its digest, never by its address.
An entry document never changes once published. To change an entry, publish a new document, whose path has a new digest, and point the index line at it. Keep the old document if anyone may have pinned it.
The index¶
The index, altar-registry/1, names the registry and lists its entries. A listing and its filters read only the
index, so each line holds what a listing shows and nothing more.
| Key | Required | What it is |
|---|---|---|
format |
yes | "altar-registry/1" |
name |
yes | The registry's name. A user can add it under another name. |
description |
no | One sentence for altar resources. |
entries |
yes | The index lines, below. |
Each line of entries:
| Key | Required | What it is |
|---|---|---|
id |
yes | The entry's id, unique in the registry ignoring case, at most 128 characters: one or more segments of letters, digits, ., _ and -, joined by single /. No segment is empty, . or .., and the id neither starts nor ends with /. Users type it after the registry's name, such as lab/ENCSR296UHQ or lab/k562/dnase: the registry is the name's first segment, and the id the rest. A model listed under it is keyed <registry>-<id> in lowercase, each / written - and each . written _, such as lab-k562-dnase; two ids that would share a key cannot be listed in one run. |
aliases |
no | For a genome, other names that name it as its id does, such as hg38; unique with every id and alias of the registry, ignoring case. A model entry has none. |
name |
yes | The entry's display name, which a listing leads with. |
type |
yes | The model type a binding registers, such as CHROMBPNET, or "GENOME". |
plugin_id |
for a model | The binding's plugin id, such as org.kundajelab.altar.model.chrombpnet. |
training_reference |
no | {genome_build, digest}: the genome a model was trained on, and its FASTA's digest. For a genome, the genome itself. |
size_bytes |
no | What the entry's files take once fetched. |
facets |
no | The metadata a listing filters on (see Metadata). |
entry |
one of entry and document |
The entry document's path, relative to the index, ending in sha256/<hex>.json. |
document |
one of entry and document |
The entry document itself, inline, for a small registry shipped in a package. |
page |
no | A web page about the entry: an https:// URL without a credential. |
{
"format": "altar-registry/1",
"name": "lab",
"description": "The lab's ChromBPNet models",
"entries": [
{
"id": "ENCSR296UHQ",
"name": "ChromBPNet, K562 DNase-seq",
"type": "CHROMBPNET",
"plugin_id": "org.kundajelab.altar.model.chrombpnet",
"training_reference": {
"genome_build": "hg38",
"digest": "sha256:9cce8b926416dd96b152deea85188495b75f7ac8d634cc723a017067be8702b7"
},
"facets": {
"assay": "DNase-seq",
"cell_types": ["K562"],
"organism": "Homo sapiens",
"project": "ENCODE"
},
"entry": "entries/sha256/b3ce2b2c41d3a947b5bfc481e946b6afde98cadc955c4bdfe410b0484f2e3697.json",
"page": "https://www.encodeproject.org/ENCSR296UHQ/"
}
]
}
This index lists ENCODE's K562 DNase-seq ChromBPNet model, trained on NCBI's GRCh38 no-alt analysis set,
genome/grch38/ncbi/no-alt-analysis-set, whose FASTA digest training_reference gives. The hexadecimal digits in entry are the SHA-256 of the entry document's bytes: here,
of the example in A model's configuration, saved exactly as shown with a final newline.
Altar checks the document it fetches against them, so the index pins every entry it lists. An inline document's
digest is taken over its canonical JSON: keys sorted, no spaces, UTF-8.
A relative URI, in the index or in any entry, is relative to the index: entries/sha256/<hex>.json and
files/weights.h5 are beside registry.json, however deep the document that names them.
An entry¶
An entry document, altar-registry-entry/1, holds everything a run needs: a model's configuration, or a
genome's FASTA, with every file named by its digest.
| Key | Required | What it is |
|---|---|---|
format |
yes | "altar-registry-entry/1" |
id, type |
yes | The same as the index line's. |
name, description |
no | As a listing shows them. |
configuration |
for a model | The model's saved configuration (below). |
prepared |
no | Files the model's preparation step writes, so a run need not run altar prepare (see Prepared files). |
reference |
with prepared |
The genome the model was trained on and its prepared files made from: {genome_build, genome: {digest}, genome_index: {digest}}, the digests of its FASTA and of the FASTA's .fai. |
genome |
for a genome | The genome's FASTA and contigs (see A genome entry). |
license |
no | The terms the files are published under, or where they are stated. |
citation |
no | How to cite the model or genome. |
page |
no | A web page about the entry, such as its page in a catalog: an https:// URL without a credential. altar resources REGISTRY/ID shows it, or the index line's page when the entry gives none. |
metadata |
no | What the entry describes (see Metadata). |
upstream |
no | [{url, description, source_checksum}]: where the files first came from, recorded for provenance. Altar never fetches it. |
A model's configuration¶
configuration is the saved configuration a binding writes with save_configuration, and reads back with
load_configuration:
| Key | What it is |
|---|---|
format |
"altar.model-configuration" |
plugin_id, schema_id, schema_version |
The binding and the version of its configuration schema. A newer binding migrates an older version. |
plugin_version |
Optional: the binding's version that wrote it. |
configuration |
The binding's configuration fields. |
Every object in configuration.configuration that has a uri is a file reference: Altar fetches it, and before
it hands the configuration to the binding it replaces the uri with the path of the fetched copy and removes the
keys only a fetch reads (member, source_checksum, size_bytes, compression). The binding and its runtime
see local files. Its identity reads each file's digest, never its uri, so the scores are the same, and are
reused, whether a run read the files from a registry or from anywhere else.
This is the entry of ENCODE's K562 DNase-seq ChromBPNet model. It takes its five folds and its peaks from the two archives ENCODE publishes, as archive members, each with the member's digest and the archive's:
{
"format": "altar-registry-entry/1",
"id": "ENCSR296UHQ",
"type": "CHROMBPNET",
"name": "ChromBPNet, K562 DNase-seq",
"configuration": {
"format": "altar.model-configuration",
"plugin_id": "org.kundajelab.altar.model.chrombpnet",
"schema_id": "org.kundajelab.altar.model.chrombpnet.configuration",
"schema_version": "0.0.3",
"configuration": {
"weights": [
{
"uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
"member": "fold_0/model.chrombpnet_nobias.fold_0.ENCSR000EOT.h5",
"digest": "sha256:920780b6855e45efa9c62bb00b79f60ed21b64c371fb01145cf7f83468cb8bfd",
"source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
},
{
"uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
"member": "fold_1/model.chrombpnet_nobias.fold_1.ENCSR000EOT.h5",
"digest": "sha256:26fa44f7f356f89aa55bfe24e71c9d84ca10187cd211b6db57a732d2146d4d7e",
"source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
},
{
"uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
"member": "fold_2/model.chrombpnet_nobias.fold_2.ENCSR000EOT.h5",
"digest": "sha256:85f352becdadcbffd640fd06e1dd0fc5025b15dc704db105b151ebe981c5643d",
"source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
},
{
"uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
"member": "fold_3/model.chrombpnet_nobias.fold_3.ENCSR000EOT.h5",
"digest": "sha256:95a70fbca9197f7ded8bba430d3d47db5484a4b68240fc8d854efafecc210454",
"source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
},
{
"uri": "https://www.encodeproject.org/files/ENCFF574YLK/@@download/ENCFF574YLK.tar.gz",
"member": "fold_4/model.chrombpnet_nobias.fold_4.ENCSR000EOT.h5",
"digest": "sha256:e05929e0443c9fb9955b395ee09e5b1995cf696ca7702e074d4d242a9e11db6f",
"source_checksum": {"sha256": "7ac3a6f1fa8e39d4f7890d048b0215c53a1068a9f4503f9989e1ae79e62714c5"}
}
],
"peaks": {
"uri": "https://www.encodeproject.org/files/ENCFF926SHM/@@download/ENCFF926SHM.tar.gz",
"member": "peaks.all_input_regions.ENCSR000EOT.bed.gz",
"digest": "sha256:830c201f4ff43b3080e1b319a89f78a52f9fea7b931824ce2a05a2ee5243db0a",
"source_checksum": {"sha256": "ea1a1061c887988b1d29b8ad3fad7b39899a8457f010258a22f395ad39784546"}
},
"training_genome_build": "hg38",
"image": "riyavsinha/altar-chrombpnet@sha256:935bdc0e20bbeaa04c10bfeb8a705a6aecea49b66742b81d47ef2dcfe120d4ee"
}
},
"reference": {
"genome_build": "hg38",
"genome": {"digest": "sha256:9cce8b926416dd96b152deea85188495b75f7ac8d634cc723a017067be8702b7"},
"genome_index": {"digest": "sha256:502a1b8fb73ccd53285c28a0f12df90c818b4fe3de1e862ef47c593ef1a0a4b4"}
},
"license": "ENCODE data, free to use; https://www.encodeproject.org/help/citing-encode/ says how to cite them",
"citation": "Pampari A et al. ChromBPNet: bias factorized, base-resolution deep learning models of chromatin accessibility reveal cis-regulatory sequence syntax, transcription factor footprints and regulatory variants. bioRxiv (2024). doi:10.1101/2024.12.25.630221",
"metadata": {
"assay": "DNase-seq",
"cell_types": ["K562"],
"organism": "Homo sapiens",
"project": "ENCODE"
},
"upstream": [
{
"url": "https://www.encodeproject.org/ENCSR296UHQ/",
"description": "The ENCODE model dataset: ENCFF574YLK holds the five folds, ENCFF926SHM the peaks"
},
{
"url": "https://www.encodeproject.org/experiments/ENCSR000EOT/",
"description": "The ENCODE DNase-seq experiment the model was trained on"
}
]
}
A genome entry¶
A genome entry has type = "GENOME" and holds genome in place of configuration:
| Key | Required | What it is |
|---|---|---|
genome_build |
yes | The assembly, such as hg38 or mm10. A run's [run] genome_build must name the same one. |
fasta |
yes | The FASTA, a file reference. |
fai |
no | {digest, uri}: the FASTA's .fai index. Without uri, Altar writes the index and checks it against digest. |
contigs |
no | Every contig of the FASTA, in order: {name, canonical, length, digest}, with the name a variant key gives it and its GA4GH refget digest. A fetch checks the index against them. |
collections |
no | {seqcol, canonical}: GA4GH sequence-collection level-0 digests of the contigs, and of their canonical names. |
source |
no | Who publishes the FASTA: {authority, accession, product, file_name, url, description}. authority is NCBI, UCSC or ENCODE; accession the assembly accession, such as GCA_000001405.15; product the publisher's own name for the file; url its https:// page; and description one line on what sets this file apart from others of its assembly. authority, product and description are required. |
Altar's own genomes are named <assembly>/<authority>/<product>, such as grch38/ucsc/hg38: GRC's name for the
assembly, who publishes the file, and the publisher's own name for it.
{
"format": "altar-registry-entry/1",
"id": "grch38/ucsc/hg38",
"type": "GENOME",
"name": "Human GRCh38, UCSC hg38.fa",
"genome": {
"genome_build": "hg38",
"source": {
"authority": "UCSC",
"accession": "GCA_000001405.15",
"product": "hg38",
"file_name": "hg38.fa.gz",
"url": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/",
"description": "Initial GRCh38 release with its 261 alt contigs and no fix patches; soft-masked (repeats in lowercase); ambiguity codes written as N"
},
"fasta": {
"uri": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz",
"compression": "gzip",
"digest": "sha256:5be01555d98347fdb3714dc84c6f77c9d8bc774adcf32c6f7a8fa06f5baf5e51",
"size_bytes": 3273481150,
"source_checksum": {"sha256": "c1dd87068c254eb53d944f71e51d1311964fce8de24d6fc0effc9c61c01527d4"}
},
"fai": {"digest": "sha256:3b425de206296a5c8053023fa5ca61da43cfe78c1737c12e58c83367c7e83c21"},
"contigs": [
{"name": "chr1", "canonical": "chr1", "length": 248956422, "digest": "SQ.2YnepKM7OkBoOrKmvHbGqguVfF9amCST"}
]
}
}
A run that names a genome, [variants] reference = "hg38", looks it up in the genome registry, by id or by
alias. A model entry's
reference names its genome by digest, so Altar can say which genome entry it is.
File references¶
Every file an entry names, in its configuration, its prepared files or its genome, takes one of two forms.
A direct copy, {uri, digest}: the bytes at uri are the file.
An archive member, {uri, member, digest, source_checksum}: uri is a tar, tar.gz or zip archive, and
member is the path of the file inside it. Use it to point at the archive a publisher first released, such as
ENCODE's model tarballs, instead of hosting a copy.
| Key | Required | What it is |
|---|---|---|
uri |
yes | https://, gs://, s3://, hf://, or a path relative to the index, such as files/weights.h5, in every document. |
digest |
yes | sha256: and 64 lowercase hexadecimal digits: the digest of the file Altar keeps, the member itself for an archive member, unpacked for a compressed copy. |
member |
for an archive member | The member's path inside the archive. |
source_checksum |
no | The publisher's checksum of the bytes at uri, such as {"md5": "…"} or {"sha256": "…"} (plain hexadecimal; md5, sha1, sha256 or sha512). Checked as the bytes arrive. |
compression |
no | "gzip": the bytes at uri are a gzip stream of the file, unpacked as they arrive. Not with member. |
size_bytes |
no | The size of the file Altar keeps. A fetch stops a download that grows past it; give it for every large file. |
revision |
no | A provider revision, a string or null, as an Altar saved configuration's file references hold it. Identity reads only digest. |
A fetch downloads each archive once, however many members an entry takes from it, extracts the members it needs
and checks each against its digest. It deletes the archive afterwards unless --keep-archives is given. A file
whose bytes do not match its digest, because the publisher replaced it at the same address, stops the fetch with
an error that names the entry, the URL and both digests. Altar never accepts it.
A download also stops once it is larger than the file's size_bytes, or, for a file without one and for an
archive, than 64 GB, which the reader's ALTAR_FETCH_MAX_BYTES changes. Give size_bytes for every large file. A
gzip file is unpacked a megabyte at a time and stops at the same limit, so a small download cannot unpack into a
disk-filling one. In an archive, a member stored as a hard link to an earlier member is read from that member; a
symbolic link is refused. Members are read into the store by digest and never written under their names, so no
name in an archive reaches the reader's files.
digest names the file in both forms, so you can move a file from an archive member to a direct copy, or between
hosts, without changing any identity: runs that read the old address reuse scores computed from the new one.
Every file needs its digest before it is published. An archive member whose digest you do not know yet cannot be listed: extract it once and hash it.
Metadata¶
metadata describes what an entry models. Every key is optional, and you may add others:
| Key | Value |
|---|---|
assay |
one value |
cell_types, organs, systems, life_stages |
a list of values |
sex, biosample, treatment, organism, source |
one value |
project |
one value, such as "ENCODE" |
A value is an ontology term, {"id": "CL:0000084", "name": "T cell"}, or plain text when no term applies.
An index line's facets repeat the values of assay, cell_types, organs, systems, life_stages, sex,
organism and project, so a listing filters without reading every entry. Give only those keys, with the same
values as the entry's metadata.
altar resources <registry> --where cell_types=k562 matches a facet ignoring case, and a term by its id or its
name. Repeated --where options must all match. --search dnase k562 matches the words of an entry's name and of
its facets' names, each word as a prefix.
Prepared files¶
Some models read files a preparation step writes, such as ChromBPNet's calibration files, which altar prepare
publishes under models/<model key>/ in the cache. An entry that lists them in prepared saves every user that
step: a run puts them where altar prepare would have, and scores without it.
| Key | What it is |
|---|---|
path |
Where altar prepare publishes the file, relative to models/<model key>/. |
resource |
The file, a file reference. |
"prepared": [
{
"path": "preprocessing.manifest.json",
"resource": {
"uri": "https://example.org/lab/my-model/preprocessing.manifest.json",
"digest": "sha256:…"
}
}
]
The binding checks prepared files as it checks the ones altar prepare publishes, so they must be the files
preparation would have written for this configuration and reference.
Digests and pinning¶
A digest is the SHA-256 of a file's bytes. Altar uses three kinds:
- A file's
digestidentifies the file. It enters the identity of the model's scores, so two runs that fetch the same files reuse each other's scores, wherever the files were hosted. - An entry's digest, the hexadecimal digits in its path, identifies the entry document. A user who writes
resources = "lab/ENCSR296UHQ@sha256:<hex>"gets exactly that document, even after the index points elsewhere, as long as it is still published or already fetched. source_checksumis the publisher's checksum of the bytes at a URI, and only checks a download.
A run records the registry's name and address, the entry's id and the entry's digest in its provenance. None of them enters identity.
Rules a check enforces¶
The JSON Schemas below state each document's shape. altar resources check also checks what a schema cannot:
- Each entry path ends in
sha256/<hex>.json, and the document there has that digest. - Each document parses, and its
idandtypeare its index line's. Ids and aliases are unique ignoring case, and only a genome has aliases. - A model entry's
plugin_idis its configuration's. When the binding is installed, the configuration loads. training_referenceagrees with the entry'sreference, andfacetswith itsmetadata.- Every file has a digest. Every URI is
https://,gs://,s3://,hf://, or relative; neverhttp://, a local path, or a URL with a credential (a user name or password, or a signed-URL or token query parameter). Users pass credentials through a connection.
A fetch holds a registry hosted elsewhere to the same rules, its own address included, after it resolves each
relative URI against the index, and refuses anything else before reading it. It also connects only to public
hosts. A host written as an IP address is refused, and so is a host whose name points at the reader's machine, a
private network or a cloud metadata service; the address is checked when the fetch connects, so a name cannot
point somewhere else between the check and the download. A redirect is checked as a new address, may not leave
https://, and a fetch follows at most five. A fetch uses no HTTP proxy. Only a registry on the reader's own disk
may name local files or http://.
A gs://, s3:// or hf:// file is read anonymously unless the reader names a connection for the registry, so
publish files that anyone can read. A registry on an intranet, whose hosts are on a private network, needs its
readers' opt-in: allow_private_hosts = true on its [registries] table or on the connection it reads through.
Check a registry¶
Run the check on the index before you publish it, and in your publishing CI:
It reads the index and every entry it names, but downloads no other file. It prints each problem on its own line
and exits with status 2 if there is one, or 0 if there is none. --json prints the problems as JSON. In Python,
altar.testing.check_registry(uri) returns the same list of problems.
Host it¶
Put registry.json and entries/ in one directory, and publish it anywhere Altar reads:
| Where | A user adds it as |
|---|---|
| A web server, such as GitHub Pages | lab = "https://example.org/registry/registry.json" |
| A Google Cloud Storage bucket | lab = "gs://lab-models/registry.json" |
| A Hugging Face repository | lab = "hf://datasets/lab/models/registry.json" |
| A directory on a shared disk | lab = "/shared/registries/lab/registry.json" |
under [registries] in a run spec. Without a connection, a gs://, s3:// or hf:// registry and the files
it names are read anonymously, never with the reader's own cloud credentials, so a public bucket or repository
works as it is. A private one is read with a connection:
lab = { uri = "gs://lab-models/registry.json", connection = "lab" }, and the run spec's [connections.lab]
gives its options. Never put a credential in the registry itself.
A registry on an intranet, whose hosts are on a private network, needs the reader's opt-in:
lab = { uri = "https://models.lab.internal/registry.json", allow_private_hosts = true }.
Ship a registry in a package¶
A binding can ship a registry with its package. Put the index, with entries inline or beside it, in the package,
and register a PackagedRegistry subclass under the altar.registries entry-point group:
from pathlib import Path
from altar.registries import PackagedRegistry
class LabRegistry(PackagedRegistry):
name = "lab"
@classmethod
def index_path(cls) -> Path:
return Path(__file__).with_name("registry.json")
The entry point's name is the registry's name. Once the package is installed, altar resources lists it.
JSON Schemas¶
altar-registry-1.schema.json: the index.altar-registry-entry-1.schema.json: an entry document.