Skip to content

Model and genome registries

altar.registries reads registries of models and reference genomes: their index and entry documents, the cache a fetch fills, and the check a publisher runs. altar resources lists, fetches and checks them. Find and fetch models and genomes shows the commands and the run-spec keys, and Publish a registry specifies the format.

Provisional: every name in this module is provisional. See Provisional names.

A package ships a registry with a PackagedRegistry subclass registered under the altar.registries entry-point group. The entry point's name is the registry's name, and index_path returns the packaged index:

[project.entry-points."altar.registries"]
lab = "lab_models.registry:LabRegistry"
from pathlib import Path
from typing import ClassVar

from altar.registries import PackagedRegistry


class LabRegistry(PackagedRegistry):
    name: ClassVar[str] = "lab"

    @classmethod
    def index_path(cls) -> Path:
        return Path(__file__).with_name("registry.json")

registries_from returns the registries a command can name: the installed ones, then those of ALTAR_REGISTRIES, then a run spec's [registries]. fetch_entry fetches one entry into a cache, and fetched_entry reads one back from the cache without downloading anything, as a run does:

from pathlib import Path

from altar.registries import EntryName, fetch_entry, fetched_entry, registries_from

registries = registries_from()
fetch_entry(registries, EntryName("legnet", "wtc11"), Path(".altar"))
entry = fetched_entry(registries, EntryName("legnet", "wtc11"), Path(".altar"))
print(entry.saved_configuration()["configuration"]["weights"]["uri"])  # the checkpoint's path in the cache

altar.testing.check_registry and altar.testing.assert_valid_registry check a registry from a test suite.

registries

Registries of models and reference genomes that anyone can host, and the cache a fetch fills (provisional).

A registry is an index, altar-registry/1, that lists entries, each an altar-registry-entry/1 document: a model's saved configuration and the files it names, or a genome's FASTA. RegistryIndex and RegistryEntry are their schemas, and registry_json_schema and entry_json_schema the published JSON Schemas. Every file is named by its SHA-256 digest, as a direct copy or as a member of an archive (FileReference).

Installed packages ship registries through the altar.registries entry-point group, each a PackagedRegistry; core ships genome. ALTAR_REGISTRIES and a run spec's [registries] add others (registries_from). altar resources fetch <registry>/<id> calls fetch_entry, which caches the index and the entry under <cache>/registries/<name>/ and puts every file in the content-addressed store, <cache>/resources/sha256/. A run names an entry, [models.<name>] resources = "legnet/wtc11" or [variants] reference = "hg38", and reads it through fetched_entry, which never downloads. check_registry validates a published registry.

A registry hosted elsewhere may name only https://, gs://, s3:// and hf:// files on public hosts, and a fetch refuses anything else with RegistryError or UnsafeAddressError. Without a connection, cloud files are read anonymously. allow_private_hosts (RegistryLocation) admits an intranet registry. A download that grows past its file's declared size, or past ALTAR_FETCH_MAX_BYTES (64 GB), stops with SizeLimitError.

Every name here is provisional: the format is a draft, and the bindings that still ship their own fetch commands will move onto it.

GENOME_REGISTRY module-attribute

GENOME_REGISTRY = 'genome'

The registry [variants] reference names a genome in.

WHERE_KEYS module-attribute

WHERE_KEYS = (
    "name",
    "id",
    "type",
    "genome_build",
    *FACET_KEYS,
)

The keys altar resources <registry> --where filters on.

EntryName dataclass

EntryName(registry: str, id: str, pin: str | None = None)

A name that names an entry: <registry>/<id>, optionally pinned to an entry digest.

The registry is the name's first segment and the id the rest, which may hold / itself: genome/grch38/ncbi/no-alt-analysis-set is the entry grch38/ncbi/no-alt-analysis-set of genome.

model_key property

model_key: str

The [models.<key>] key a run gives the model this entry names: <registry>-<id>, lowercase.

Each / in the id becomes -, as the one after the registry does, and each . becomes _, so the key is a bare TOML key: lab/ENCSR296UHQ is lab-encsr296uhq, lab/v1.2 is lab-v1_2, and lab/k562/dnase is lab-k562-dnase. It ignores the pin and the entry's display name, so it is the same for every document of the entry and across a rename. It is a valid model key only for a registry name and id that follow a registry's rules (entry_name_problem), which a run checks before it uses the key; two such entries may still share one (lab/v1.2 and lab/v1_2, or lab/a/b and lab/a-b), which a run refuses.

FetchedEntry dataclass

FetchedEntry(
    source: RegistrySource,
    line: IndexEntry,
    entry: RegistryEntry,
    entry_sha256: str,
    paths: Mapping[str, Path],
    cache_dir: Path,
)

An entry whose files are all in the cache: where each one is, and what a run builds from them.

paths instance-attribute

paths: Mapping[str, Path]

Each file's digest, and its path in the store.

cache_dir instance-attribute

cache_dir: Path

The cache the store is in.

name property

name: str

The entry's name, <registry>/<id>, spelt as the registry spells them.

fasta property

fasta: Path

A genome entry's FASTA in the store; its .fai is beside it.

saved_configuration

saved_configuration() -> dict[str, Any]

Return the entry's saved configuration with each file's uri its path in the store.

The fetch keys (member, source_checksum, size_bytes, compression) are removed, so the binding reads the configuration it saved. Identity reads digests, so this changes no lineage.

prepared

prepared() -> dict[str, Path]

Return each prepared file's path relative to the model's directory, and its path in the store.

provenance

provenance() -> dict[str, str]

Return what a run records about where this entry came from: never part of identity.

FetchSummary dataclass

FetchSummary(
    name: str,
    type: str,
    entry_sha256: str,
    files_fetched: int,
    files_present: int,
    bytes_fetched: int,
    files: dict[str, str],
)

What fetch_entry did: the entry, its digest, and the files it fetched or found already in the cache.

files instance-attribute

files: dict[str, str]

Where the entry names each file, and its path in the cache.

NotFetchedError

Bases: RegistryError

A run names an entry, or a registry's index, that is not in the cache; the message names the command.

PackagedRegistry

A registry an installed package ships; a distribution registers one under altar.registries.

The entry point's name must equal name, the registry's name. index_path returns the packaged index, an altar-registry/1 file whose entries are inline or in entries/sha256/ beside it.

index_path classmethod

index_path() -> Path

Return the path of the packaged index.

Registries dataclass

Registries(
    sources: tuple[RegistrySource, ...],
    failures: tuple[PluginDiscoveryFailure, ...] = (),
)

The registries a command or run can name, and the installed ones that failed to load.

require

require(name: str) -> RegistrySource

Return the registry called name, or raise RegistryError with the closest names.

RegistryError

Bases: LookupError

A name that names no registry or entry, or a registry that cannot be read; the message says what to do.

RegistryLocation

Bases: NamedTuple

Where a run spec's [registries] puts a registry: its index, its connection, and its private-host opt-in.

RegistrySource dataclass

RegistrySource(
    name: str,
    uri: str,
    origin: Origin,
    distribution: str | None = None,
    connection_name: str | None = None,
    connection: Connection | None = None,
    allow_private_hosts: bool = False,
)

One registry: its name, where its index is, where that came from, and the connection that reads it.

allow_private_hosts class-attribute instance-attribute

allow_private_hosts: bool = False

Whether a registry hosted elsewhere may name hosts on a private network, as an intranet mirror does.

key property

key: str

The name as the cache spells it on disk: lowercase.

is_local property

is_local: bool

Whether the index is a file on this machine, which is read in place and never cached.

check_uri

check_uri(uri: str) -> None

Raise RegistryError unless this registry may name uri, resolved against its index.

A registry on this machine may name anything; one hosted elsewhere only what remote_uri_problem allows.

Contig

Bases: _Document

One contig of a FASTA: its name, the name a variant key gives it, its length and its refget digest.

EntryConfiguration

Bases: _Document

An Altar saved model configuration (altar.model-configuration), as a binding's save_configuration writes.

Every object in configuration that has a uri is a file reference, and may carry the fetch keys of a FileReference (member, source_checksum, size_bytes, compression), which the run removes before it gives the configuration to the binding.

Facets

Bases: _Document

The metadata an index line repeats so altar resources <registry> --where filters without reading entries.

FileReference

Bases: _Document

One file an entry names: where it is published and the SHA-256 digest of the file Altar keeps.

A direct copy is {uri, digest}: the bytes at uri are the file, or with compression = "gzip" a gzip stream of it. An archive member is {uri, member, digest}: uri is a tar, tar.gz or zip archive and member the path of the file inside it. digest is the file's digest in both forms, so identity does not depend on the form. source_checksum is the publisher's checksum of the bytes at uri (the archive, or the compressed download), checked when the bytes arrive.

Genome

Bases: _Document

What a genome entry holds: its assembly, its FASTA, the FASTA's index, its contigs, and who publishes it.

contigs lists every contig in FASTA order. source says who publishes the FASTA and what sets it apart.

GenomeSource

Bases: _Document

Who publishes a genome's FASTA, under what name, and what sets that file apart from others of its assembly.

authority is the publisher; accession the assembly accession the publisher gives the file, such as GCA_000001405.15; product the publisher's own name for the file, such as hg38 or male.hg19; file_name the name it publishes the file under, and url the page it describes it on. description is one line on what makes the product differ from the others of its assembly.

IndexEntry

Bases: _Document

One line of an index: what a listing shows and filters on, and where the entry document is.

entry is the entry document's path relative to the index, ending in sha256/<hex>.json, where <hex> is the SHA-256 of the document's bytes. A packaged registry may give the document inline, as document, instead; its digest is then taken over its canonical JSON (sorted keys, no spaces, UTF-8).

entry_sha256 property

entry_sha256: str

The entry document's digest: from entry's path, or of an inline document's canonical JSON.

names

names() -> list[str]

Return the id and every alias, as the index spells them.

Metadata

Bases: BaseModel

What an entry describes, in a host catalog's public vocabulary. Every key is optional; others may be added.

assay is single; cell_types, organs, systems and life_stages are lists. A value is an ontology term {id, name} or plain text.

PreparedFile

Bases: _Document

A file the model's preparation step writes, published so a run need not prepare it.

path is where altar prepare would publish it, relative to the model's directory, models/<model key>/.

Reference

Bases: _Document

The genome a model was trained on, and its prepared files made from: its assembly, and its files' digests.

genome is the FASTA's digest and genome_index its .fai's. A genome registry entry with the same FASTA digest names the same genome.

RegistryEntry

Bases: _Document

An entry document, altar-registry-entry/1: one model, or one genome, with every file it names.

A model entry holds a saved configuration, and optionally the prepared files its preparation step writes and the reference genome it was trained on. A genome entry (type = "GENOME") holds genome instead. Both may give a license, a citation, a web page about the entry, as its index line may, metadata and the upstream sources of their files.

files

files() -> list[FileReference]

Return every file a fetch puts in the cache: the configuration's, then the prepared files, or the FASTA.

RegistryIndex

Bases: _Document

An index, altar-registry/1: the registry's name and description, and one line per entry.

get

get(entry_id: str) -> IndexEntry | None

Return the line whose id or alias is entry_id, ignoring case.

Term

Bases: _Document

An ontology term: its identifier, such as CL:0000236, and its label.

TrainingReference

Bases: _Document

An index line's summary of the genome an entry is on: its assembly and its FASTA's digest.

Upstream

Bases: _Document

Where an entry's files first came from, recorded for provenance; Altar never fetches it.

UnsafeAddressError

Bases: OSError

An address a registry hosted elsewhere names, or a host it resolves to, that a fetch may not reach.

FetchProgress dataclass

FetchProgress(
    stage: Literal[
        "start", "downloading", "fetched", "indexing"
    ],
    entry: str,
    files_total: int,
    bytes_total: int,
    files_present: int = 0,
    bytes_present: int = 0,
    files_done: int = 0,
    bytes_done: int = 0,
    path: str | None = None,
    file_bytes: int = 0,
    file_size: int | None = None,
    present: bool = False,
    archive: str | None = None,
)

How far a fetch has got, as its progress callback receives it. Sizes are of the stored files.

stage is "start" once; "downloading" at most every 30 seconds while a file or an archive arrives, with file_bytes so far and archive set for an archive; "fetched" after each file is in the store, or found already there (present); and "indexing" before a genome's FASTA is indexed.

SizeLimitError

Bases: ValueError

A download larger than its entry says its file is, or than the fetch's limit.

check_registry

check_registry(
    uri: str,
    *,
    connection: Connection | None = None,
    allow_private_hosts: bool = False,
) -> list[str]

Return every problem with the registry whose index is at uri, and with each entry it names; empty if none.

It reads the index and every entry document, and no other file. Each document must parse, each entry's bytes must have the digest its path gives, and each index line must agree with its entry. Every URI must be one a published registry may name (altar.plugins.registries.documents.uri_problem with publishing). When the entry's binding is installed, its configuration must load. An index hosted elsewhere is read as a fetch reads it, and its entries are read only where a fetch would read them (RegistrySource.check_uri).

fetch_entry

fetch_entry(
    registries: Registries,
    name: EntryName,
    cache_dir: str | PathLike[str],
    *,
    progress: Callable[[FetchProgress], None] | None = None,
    keep_archives: bool = False,
    refresh: bool = False,
    stop: Callable[[], bool] | None = None,
) -> FetchSummary

Fetch the entry name names and every file it names into the cache; see the module docstring.

An id the cached index does not list makes the fetch read the index again once. A genome's FASTA also gets its .fai, checked against the entry's digest and contigs. stop is asked before every read of a download, each at most a megabyte or the 60 seconds an https:// read waits; when it returns true the fetch raises FetchStopped, as altar resources fetch --jobs stops the fetches beside one that failed.

fetched_entry

fetched_entry(
    registries: Registries,
    name: EntryName,
    cache_dir: str | PathLike[str],
    *,
    command: Callable[[EntryName, RegistrySource], str]
    | None = None,
) -> FetchedEntry

Return the fetched entry name names, reading only the cache and local registries; nothing is downloaded.

Raises RegistryError for an unknown registry or entry, and NotFetchedError, with command's fetch command, for an index, entry or file not in the cache, or a file that has changed since it was fetched.

get_registry_source_registry

get_registry_source_registry() -> PluginRegistry[
    type[PackagedRegistry]
]

Return the registry of packaged registries (the altar.registries entry-point group).

installed_registries

installed_registries() -> Registries

Return the registries installed packages ship, setting aside entry points that fail to load.

load_index

load_index(
    source: RegistrySource,
    cache_dir: str | PathLike[str],
    *,
    refresh: bool = False,
    offline: bool = False,
    command: str = "altar resources",
) -> CachedIndex

Return the registry's index: read in place for a local registry, else from its cache, refreshed if asked.

A remote index is fetched when refresh is set, when the cache has none, or when the cached one came from another URI. With offline, nothing is fetched and a missing index raises NotFetchedError, naming command.

matches

matches(
    line: IndexEntry,
    where: Sequence[tuple[str, str]] = (),
    search: Sequence[str] = (),
) -> bool

Return whether an index line matches every where pair and every search word.

A where pair matches a value ignoring case, and an ontology term by its id or its name. A search word matches the start of a word of the line's name, id or facet names.

parse_entry_name

parse_entry_name(
    text: str,
    *,
    default_registry: str | None = None,
    registries: Registries | None = None,
) -> EntryName

Read <registry>/<id>[@sha256:<hex>]: the registry is the first segment, and the id the rest.

With default_registry, a bare <id> is in it, and so, when registries is given, is a name whose first segment names none of them: grch38/ncbi/no-alt-analysis-set is in genome, as is hg38, while lab/hg38-custom is in lab when lab is a registry. Raises ValueError for anything else.

registries_from

registries_from(
    spec: Mapping[
        str, RegistryLocation | tuple[str, str | None]
    ]
    | None = None,
    *,
    connections: Mapping[str, Connection] | None = None,
    base: str | PathLike[str] | None = None,
    environment: str | None = None,
) -> Registries

Return the installed registries, then ALTAR_REGISTRIES's, then spec's, each replacing a namesake.

spec maps a name to its index URI and the name of the connection that reads it, as a run spec's [registries] gives them. connections holds the built connections by name; a registry whose connection is missing from it reads without one, which suffices for the cache. A relative local path is read from base.

entry_json_schema

entry_json_schema() -> dict[str, Any]

Return the published JSON Schema of an entry document, altar-registry-entry/1.

registry_json_schema

registry_json_schema() -> dict[str, Any]

Return the published JSON Schema of an index, altar-registry/1.

content_path

content_path(
    cache_dir: str | PathLike[str], digest: str
) -> Path

Return where the store keeps the file whose digest is digest: <cache>/resources/sha256/<hex>.