Model and genome registries¶
altar.registries reads registries of models and reference genomes: their index and entry documents, the
cache a fetch fills, and the check a publisher runs. altar resources lists, fetches and checks them.
Find and fetch models and genomes shows the commands and the run-spec keys, and
Publish a registry specifies the format.
Provisional: every name in this module is provisional. See Provisional names.
A package ships a registry with a PackagedRegistry subclass registered under the altar.registries
entry-point group. The entry point's name is the registry's name, and index_path returns the packaged index:
from pathlib import Path
from typing import ClassVar
from altar.registries import PackagedRegistry
class LabRegistry(PackagedRegistry):
name: ClassVar[str] = "lab"
@classmethod
def index_path(cls) -> Path:
return Path(__file__).with_name("registry.json")
registries_from returns the registries a command can name: the installed ones, then those of
ALTAR_REGISTRIES, then a run spec's [registries]. fetch_entry fetches one entry into a cache, and
fetched_entry reads one back from the cache without downloading anything, as a run does:
from pathlib import Path
from altar.registries import EntryName, fetch_entry, fetched_entry, registries_from
registries = registries_from()
fetch_entry(registries, EntryName("legnet", "wtc11"), Path(".altar"))
entry = fetched_entry(registries, EntryName("legnet", "wtc11"), Path(".altar"))
print(entry.saved_configuration()["configuration"]["weights"]["uri"]) # the checkpoint's path in the cache
altar.testing.check_registry and altar.testing.assert_valid_registry check a registry from a test suite.
registries
¶
Registries of models and reference genomes that anyone can host, and the cache a fetch fills (provisional).
A registry is an index, altar-registry/1, that lists entries, each an altar-registry-entry/1 document: a
model's saved configuration and the files it names, or a genome's FASTA. RegistryIndex and RegistryEntry are
their schemas, and registry_json_schema and entry_json_schema the published JSON Schemas. Every file is named by
its SHA-256 digest, as a direct copy or as a member of an archive (FileReference).
Installed packages ship registries through the altar.registries entry-point group, each a PackagedRegistry;
core ships genome. ALTAR_REGISTRIES and a run spec's [registries] add others (registries_from).
altar resources fetch <registry>/<id> calls fetch_entry, which caches the index and the entry under
<cache>/registries/<name>/ and puts every file in the content-addressed store, <cache>/resources/sha256/.
A run names an entry, [models.<name>] resources = "legnet/wtc11" or [variants] reference = "hg38", and
reads it through fetched_entry, which never downloads. check_registry validates a published registry.
A registry hosted elsewhere may name only https://, gs://, s3:// and hf:// files on public hosts,
and a fetch refuses anything else with RegistryError or UnsafeAddressError. Without a connection, cloud files
are read anonymously. allow_private_hosts (RegistryLocation) admits an intranet registry. A download that
grows past its file's declared size, or past ALTAR_FETCH_MAX_BYTES (64 GB), stops with SizeLimitError.
Every name here is provisional: the format is a draft, and the bindings that still ship their own fetch commands will move onto it.
GENOME_REGISTRY
module-attribute
¶
The registry [variants] reference names a genome in.
WHERE_KEYS
module-attribute
¶
The keys altar resources <registry> --where filters on.
EntryName
dataclass
¶
A name that names an entry: <registry>/<id>, optionally pinned to an entry digest.
The registry is the name's first segment and the id the rest, which may hold / itself:
genome/grch38/ncbi/no-alt-analysis-set is the entry grch38/ncbi/no-alt-analysis-set of genome.
model_key
property
¶
The [models.<key>] key a run gives the model this entry names: <registry>-<id>, lowercase.
Each / in the id becomes -, as the one after the registry does, and each . becomes _, so the
key is a bare TOML key: lab/ENCSR296UHQ is lab-encsr296uhq, lab/v1.2 is lab-v1_2, and
lab/k562/dnase is lab-k562-dnase. It ignores the pin and the entry's display name, so it is the
same for every document of the entry and across a rename. It is a valid model key only for a registry name
and id that follow a registry's rules (entry_name_problem), which a run checks before it uses the key;
two such entries may still share one (lab/v1.2 and lab/v1_2, or lab/a/b and lab/a-b), which
a run refuses.
FetchedEntry
dataclass
¶
FetchedEntry(
source: RegistrySource,
line: IndexEntry,
entry: RegistryEntry,
entry_sha256: str,
paths: Mapping[str, Path],
cache_dir: Path,
)
An entry whose files are all in the cache: where each one is, and what a run builds from them.
saved_configuration
¶
Return the entry's saved configuration with each file's uri its path in the store.
The fetch keys (member, source_checksum, size_bytes, compression) are removed, so the
binding reads the configuration it saved. Identity reads digests, so this changes no lineage.
prepared
¶
Return each prepared file's path relative to the model's directory, and its path in the store.
provenance
¶
Return what a run records about where this entry came from: never part of identity.
FetchSummary
dataclass
¶
FetchSummary(
name: str,
type: str,
entry_sha256: str,
files_fetched: int,
files_present: int,
bytes_fetched: int,
files: dict[str, str],
)
What fetch_entry did: the entry, its digest, and the files it fetched or found already in the cache.
files
instance-attribute
¶
Where the entry names each file, and its path in the cache.
NotFetchedError
¶
Bases: RegistryError
A run names an entry, or a registry's index, that is not in the cache; the message names the command.
PackagedRegistry
¶
A registry an installed package ships; a distribution registers one under altar.registries.
The entry point's name must equal name, the registry's name. index_path returns the packaged index, an
altar-registry/1 file whose entries are inline or in entries/sha256/ beside it.
Registries
dataclass
¶
Registries(
sources: tuple[RegistrySource, ...],
failures: tuple[PluginDiscoveryFailure, ...] = (),
)
The registries a command or run can name, and the installed ones that failed to load.
require
¶
require(name: str) -> RegistrySource
Return the registry called name, or raise RegistryError with the closest names.
RegistryError
¶
Bases: LookupError
A name that names no registry or entry, or a registry that cannot be read; the message says what to do.
RegistryLocation
¶
Bases: NamedTuple
Where a run spec's [registries] puts a registry: its index, its connection, and its private-host opt-in.
RegistrySource
dataclass
¶
RegistrySource(
name: str,
uri: str,
origin: Origin,
distribution: str | None = None,
connection_name: str | None = None,
connection: Connection | None = None,
allow_private_hosts: bool = False,
)
One registry: its name, where its index is, where that came from, and the connection that reads it.
allow_private_hosts
class-attribute
instance-attribute
¶
Whether a registry hosted elsewhere may name hosts on a private network, as an intranet mirror does.
is_local
property
¶
Whether the index is a file on this machine, which is read in place and never cached.
check_uri
¶
Raise RegistryError unless this registry may name uri, resolved against its index.
A registry on this machine may name anything; one hosted elsewhere only what remote_uri_problem allows.
Contig
¶
Bases: _Document
One contig of a FASTA: its name, the name a variant key gives it, its length and its refget digest.
EntryConfiguration
¶
Bases: _Document
An Altar saved model configuration (altar.model-configuration), as a binding's save_configuration writes.
Every object in configuration that has a uri is a file reference, and may carry the fetch keys of a
FileReference (member, source_checksum, size_bytes, compression), which the run removes
before it gives the configuration to the binding.
Facets
¶
Bases: _Document
The metadata an index line repeats so altar resources <registry> --where filters without reading entries.
FileReference
¶
Bases: _Document
One file an entry names: where it is published and the SHA-256 digest of the file Altar keeps.
A direct copy is {uri, digest}: the bytes at uri are the file, or with compression = "gzip" a gzip
stream of it. An archive member is {uri, member, digest}: uri is a tar, tar.gz or zip archive and
member the path of the file inside it. digest is the file's digest in both forms, so identity does
not depend on the form. source_checksum is the publisher's checksum of the bytes at uri (the archive,
or the compressed download), checked when the bytes arrive.
Genome
¶
Bases: _Document
What a genome entry holds: its assembly, its FASTA, the FASTA's index, its contigs, and who publishes it.
contigs lists every contig in FASTA order. source says who publishes the FASTA and what sets it apart.
GenomeSource
¶
Bases: _Document
Who publishes a genome's FASTA, under what name, and what sets that file apart from others of its assembly.
authority is the publisher; accession the assembly accession the publisher gives the file, such as
GCA_000001405.15; product the publisher's own name for the file, such as hg38 or male.hg19;
file_name the name it publishes the file under, and url the page it describes it on. description is
one line on what makes the product differ from the others of its assembly.
IndexEntry
¶
Bases: _Document
One line of an index: what a listing shows and filters on, and where the entry document is.
entry is the entry document's path relative to the index, ending in sha256/<hex>.json, where <hex>
is the SHA-256 of the document's bytes. A packaged registry may give the document inline, as document,
instead; its digest is then taken over its canonical JSON (sorted keys, no spaces, UTF-8).
Metadata
¶
Bases: BaseModel
What an entry describes, in a host catalog's public vocabulary. Every key is optional; others may be added.
assay is single; cell_types, organs, systems and life_stages are lists. A value is an
ontology term {id, name} or plain text.
PreparedFile
¶
Bases: _Document
A file the model's preparation step writes, published so a run need not prepare it.
path is where altar prepare would publish it, relative to the model's directory,
models/<model key>/.
Reference
¶
Bases: _Document
The genome a model was trained on, and its prepared files made from: its assembly, and its files' digests.
genome is the FASTA's digest and genome_index its .fai's. A genome registry entry with the same
FASTA digest names the same genome.
RegistryEntry
¶
Bases: _Document
An entry document, altar-registry-entry/1: one model, or one genome, with every file it names.
A model entry holds a saved configuration, and optionally the prepared files its preparation step
writes and the reference genome it was trained on. A genome entry (type = "GENOME") holds genome
instead. Both may give a license, a citation, a web page about the entry, as its index line may,
metadata and the upstream sources of their files.
files
¶
files() -> list[FileReference]
Return every file a fetch puts in the cache: the configuration's, then the prepared files, or the FASTA.
RegistryIndex
¶
Bases: _Document
An index, altar-registry/1: the registry's name and description, and one line per entry.
get
¶
get(entry_id: str) -> IndexEntry | None
Return the line whose id or alias is entry_id, ignoring case.
Term
¶
Bases: _Document
An ontology term: its identifier, such as CL:0000236, and its label.
TrainingReference
¶
Bases: _Document
An index line's summary of the genome an entry is on: its assembly and its FASTA's digest.
Upstream
¶
Bases: _Document
Where an entry's files first came from, recorded for provenance; Altar never fetches it.
UnsafeAddressError
¶
Bases: OSError
An address a registry hosted elsewhere names, or a host it resolves to, that a fetch may not reach.
FetchProgress
dataclass
¶
FetchProgress(
stage: Literal[
"start", "downloading", "fetched", "indexing"
],
entry: str,
files_total: int,
bytes_total: int,
files_present: int = 0,
bytes_present: int = 0,
files_done: int = 0,
bytes_done: int = 0,
path: str | None = None,
file_bytes: int = 0,
file_size: int | None = None,
present: bool = False,
archive: str | None = None,
)
How far a fetch has got, as its progress callback receives it. Sizes are of the stored files.
stage is "start" once; "downloading" at most every 30 seconds while a file or an archive arrives,
with file_bytes so far and archive set for an archive; "fetched" after each file is in the store,
or found already there (present); and "indexing" before a genome's FASTA is indexed.
SizeLimitError
¶
Bases: ValueError
A download larger than its entry says its file is, or than the fetch's limit.
check_registry
¶
check_registry(
uri: str,
*,
connection: Connection | None = None,
allow_private_hosts: bool = False,
) -> list[str]
Return every problem with the registry whose index is at uri, and with each entry it names; empty if none.
It reads the index and every entry document, and no other file. Each document must parse, each entry's bytes
must have the digest its path gives, and each index line must agree with its entry. Every URI must be one a
published registry may name (altar.plugins.registries.documents.uri_problem with publishing). When the
entry's binding is installed, its configuration must load. An index hosted elsewhere is read as a fetch reads
it, and its entries are read only where a fetch would read them (RegistrySource.check_uri).
fetch_entry
¶
fetch_entry(
registries: Registries,
name: EntryName,
cache_dir: str | PathLike[str],
*,
progress: Callable[[FetchProgress], None] | None = None,
keep_archives: bool = False,
refresh: bool = False,
stop: Callable[[], bool] | None = None,
) -> FetchSummary
Fetch the entry name names and every file it names into the cache; see the module docstring.
An id the cached index does not list makes the fetch read the index again once. A genome's FASTA also gets its
.fai, checked against the entry's digest and contigs. stop is asked before every read of a download,
each at most a megabyte or the 60 seconds an https:// read waits; when it returns true the fetch raises
FetchStopped, as altar resources fetch --jobs stops the fetches beside one that failed.
fetched_entry
¶
fetched_entry(
registries: Registries,
name: EntryName,
cache_dir: str | PathLike[str],
*,
command: Callable[[EntryName, RegistrySource], str]
| None = None,
) -> FetchedEntry
Return the fetched entry name names, reading only the cache and local registries; nothing is downloaded.
Raises RegistryError for an unknown registry or entry, and NotFetchedError, with command's fetch
command, for an index, entry or file not in the cache, or a file that has changed since it was fetched.
get_registry_source_registry
¶
get_registry_source_registry() -> PluginRegistry[
type[PackagedRegistry]
]
Return the registry of packaged registries (the altar.registries entry-point group).
installed_registries
¶
installed_registries() -> Registries
Return the registries installed packages ship, setting aside entry points that fail to load.
load_index
¶
load_index(
source: RegistrySource,
cache_dir: str | PathLike[str],
*,
refresh: bool = False,
offline: bool = False,
command: str = "altar resources",
) -> CachedIndex
Return the registry's index: read in place for a local registry, else from its cache, refreshed if asked.
A remote index is fetched when refresh is set, when the cache has none, or when the cached one came from
another URI. With offline, nothing is fetched and a missing index raises NotFetchedError, naming
command.
matches
¶
matches(
line: IndexEntry,
where: Sequence[tuple[str, str]] = (),
search: Sequence[str] = (),
) -> bool
Return whether an index line matches every where pair and every search word.
A where pair matches a value ignoring case, and an ontology term by its id or its name. A search word
matches the start of a word of the line's name, id or facet names.
parse_entry_name
¶
parse_entry_name(
text: str,
*,
default_registry: str | None = None,
registries: Registries | None = None,
) -> EntryName
Read <registry>/<id>[@sha256:<hex>]: the registry is the first segment, and the id the rest.
With default_registry, a bare <id> is in it, and so, when registries is given, is a name whose
first segment names none of them: grch38/ncbi/no-alt-analysis-set is in genome, as is hg38, while
lab/hg38-custom is in lab when lab is a registry. Raises ValueError for anything else.
registries_from
¶
registries_from(
spec: Mapping[
str, RegistryLocation | tuple[str, str | None]
]
| None = None,
*,
connections: Mapping[str, Connection] | None = None,
base: str | PathLike[str] | None = None,
environment: str | None = None,
) -> Registries
Return the installed registries, then ALTAR_REGISTRIES's, then spec's, each replacing a namesake.
spec maps a name to its index URI and the name of the connection that reads it, as a run spec's
[registries] gives them. connections holds the built connections by name; a registry whose connection is
missing from it reads without one, which suffices for the cache. A relative local path is read from base.
entry_json_schema
¶
Return the published JSON Schema of an entry document, altar-registry-entry/1.
registry_json_schema
¶
Return the published JSON Schema of an index, altar-registry/1.
content_path
¶
Return where the store keeps the file whose digest is digest: <cache>/resources/sha256/<hex>.