Add connections and published datasets¶
A connection type tells Altar how to reach object storage or a warehouse: which client options and credentials to use for a URI. A published dataset is a copy of a source's data that its publisher hosts, so users read it in place instead of building it. Both interfaces are Provisional: they are expected to change (see Provisional names).
When to choose a connection type or a published dataset¶
- Write a connection type when your data lives in a system the shipped
gcpandfsspectypes don't reach, or that needs options they can't express. Most object stores need no new type: anfsspecconnection passes any fsspec options. Choose where data is read from shows how a run spec uses connections. - Publish a dataset from a binding when its data is openly licensed and expensive to build, and you host a built copy at a public location.
- To read through a connection from your own source, you need neither: take a
connectionkeyword infrom_options, as Read through a named connection shows.
Add a connection type¶
Subclass Connection, imported from altar.connections. Set name to the registry key, and override the methods
the type supports:
| Member | Required? | What it is |
|---|---|---|
name |
required | The registry name, and the type of a run spec's [connections.<name>] section. |
storage_options(for_uri) |
default: raises CapabilityNotSupported |
The fsspec keyword arguments that open a URI, or a bare scheme:// prefix. Returns {} for a local path. |
bigquery_client() |
default: raises CapabilityNotSupported |
A new google.cloud.bigquery.Client, which the caller owns and closes. |
from_options(**options) |
default: passes the options to the constructor | Builds the connection from a run spec's options. Override it to refuse a credential. |
This type reaches a lab's S3-compatible object store at its own endpoint, with the credentials of a named profile
that the provider library finds on its own. Put it in a module of your own, such as lab_connection.py:
from typing import Any
from altar.connections import Connection
from altar.models import CapabilityNotSupported
class LabObjectStoreConnection(Connection):
"""Reach the lab's S3-compatible object store at its endpoint, with the ambient credentials of a named profile."""
name = "lab_s3"
def __init__(self, *, endpoint_url: str, profile: str | None = None) -> None:
self.endpoint_url = endpoint_url
self.profile = profile
def storage_options(self, for_uri: str) -> dict[str, Any]:
scheme, separator, _ = for_uri.partition("://")
if not separator or scheme in ("file", "local"):
return {} # a local path needs no credentials
if scheme != "s3":
raise CapabilityNotSupported(f"the {self.name!r} connection reaches s3:// only, not {for_uri}")
options: dict[str, Any] = {"client_kwargs": {"endpoint_url": self.endpoint_url}}
if self.profile is not None:
options["profile"] = self.profile
return options
A run spec declares it like any other connection, and a source, the run's storage or a store names it:
[connections.lab]
type = "lab_s3"
options = { endpoint_url = "https://objects.lab.example.org", profile = "research" }
The rules every connection type keeps:
- A connection says how to reach data, never which data. It never enters a source's, a model's or a cache's identity.
- A run spec is a file that is saved and shared, so it never carries a credential. The constructor here takes no
secret, so the default
from_optionsrefuses one with aTypeError. A type whose constructor accepts a secret, for your own code to pass in, overridesfrom_optionsto refuse it, without repeating the secret in the message. reprnever shows an option. The base class'sreprshows only the type and its name.- Import a provider SDK inside the method that needs it. The registry lists, resolves and builds a type without importing any provider library.
- Return a new mapping from each
storage_optionscall, the same for every URI of a scheme.
Publish a dataset from a binding¶
A binding declares its published copy by returning a PublishedDataset from its AnnotationSource.published_dataset()
classmethod. Both are imported from altar.sources. A PublishedDataset holds:
locations: a tuple ofPublishedLocationvalues, each a short name and a URI. The first is the primary, which a reader gets without naming a location. The rest are mirrors that hold the same files in the same layout.identity: the contract, release and genome build of the data. A source that serves this release declares this identity, whichever location or copy it reads.release_metadata: theParquetReleaseMetadatathe binding's builder writes into every file's footer.release_fields: the footer fields, without their prefix, that every file of the release records. They must includegenome_build, set to the identity's genome build.cluster_by: the columns a BigQuery copy is clustered on,variant_idby default. A dataset read by something else, such as a link dataset read by interval, names its own.
from altar.sources import (
AnnotationColumn,
AnnotationContract,
ParquetReleaseMetadata,
PublishedDataset,
PublishedLocation,
)
CONSERVATION = AnnotationContract(
source_id="org.example.annotation.conservation",
schema_version="1.0.0",
columns=(AnnotationColumn("conservation_score", "float", "Conservation score"),),
)
RELEASE_METADATA = ParquetReleaseMetadata(
prefix="altar_example_conservation.",
distribution="altar-example-conservation",
label="Example conservation",
command="altar-example-conservation-build",
)
PUBLISHED = PublishedDataset(
locations=(
PublishedLocation("gcs", "gs://publisher-bucket/conservation-2026-10"),
PublishedLocation("hf", "hf://datasets/publisher/conservation-2026-10"),
),
identity=CONSERVATION.identity(release="2026-10", genome_build="hg38"),
release_metadata=RELEASE_METADATA,
release_fields={"release": "2026-10", "genome_build": "hg38"},
)
The source returns it, and its from_options resolves the run spec's options with PUBLISHED.select_copy(...):
from typing import Self
from altar.sources import AnnotationSource, PublishedCopy
class ConservationSource(AnnotationSource):
name = "example_conservation"
@classmethod
def published_dataset(cls) -> PublishedDataset:
return PUBLISHED
@classmethod
def from_options(cls, *, dataset=None, location=None, connection=None) -> Self:
copy = PUBLISHED.select_copy("Example conservation", dataset=dataset, location=location, connection=connection)
uri = copy.uri if isinstance(copy, PublishedCopy) else copy.dataset
return cls(uri, storage_options=copy.storage_options)
The rest of the class is an ordinary source over the binding's Parquet layout; see
Add an annotation source. select_copy applies the same option rules and messages for
every publishing binding, and opens nothing:
- With neither
datasetnortable, it returns aPublishedCopy: the locationlocationnames, the primary by default, read anonymously ({"token": "anon"}on Cloud Storage) or through the connection. - With
dataset, it returns anOwnedDatasetCopy: the user's Parquet copy, read through the connection or with the provider's ambient credentials. - With
table, it returns anOwnedTableCopy: the user's BigQuery table, which needs a connection.releasenames the release the table holds, by default the published one. - It raises
ValueErrorfor options that name no single copy, such as bothdatasetandtable, orlocationwith either.
A binding without a BigQuery table mode takes neither table nor release in from_options, and refuses each by
name.
Each location holds the builder's output, the same files at every location: Parquet files under
<partition column>=<chromosome>/ directories, each holding the partition column and variant_id, and recording
the release in its footer. altar pull refuses a published dataset without release_metadata, because it could
not check a copy. List first the location the binding's base dependencies can read. A location's name is short and
lowercase (letters, digits, _ and -, starting with a letter), and is what users type after --from. Name a
Cloud Storage location by its gs:// URI: a partitioned dataset is found by listing its directory, which plain
HTTPS cannot do there.
A binding publishes one release, its latest, and a binding version pins the release it reads by default. Publish a dataset release is the maintainers' procedure for building, hosting and bumping a release.
Register the entry point¶
Register a connection type under the altar.connections group, with its name as the key:
A publishing binding registers its source under altar.annotation_sources as usual. altar sources, altar pull
and the run driver find its published dataset through the registry.
Test with the conformance suites¶
A connection type¶
Subclass ConnectionContract from altar.testing, and describe the type with its fixtures:
import pytest
from altar.testing import ConnectionContract
from lab_connection import LabObjectStoreConnection
class TestLabObjectStoreConnection(ConnectionContract):
@pytest.fixture
def connection_type(self):
return LabObjectStoreConnection
@pytest.fixture
def connection_options(self):
return {"endpoint_url": "https://objects.lab.example.org", "profile": "research"}
@pytest.fixture
def served_uri(self):
return "s3://lab-bucket/dataset"
The other fixtures are optional: unsupported_uri (a URI whose scheme it must decline), reaches_bigquery,
credential_options (spec options that carry a credential) and connection_holding_secret (a connection built
in code with a secret, and the secret). Override connection to install fakes for a provider SDK first.
The suite checks that the type is registered under its name; that a local path needs no options; that a served
URI gets a fresh mapping, the same for its bare scheme:// prefix; that an unserved scheme, and BigQuery when the
type does not reach it, raise CapabilityNotSupported; that from_options refuses a credential without repeating
it; that repr shows no option; that listing, resolving and building the type imports no provider library; and
that PublishedDataset.select_copy reaches each copy through it without changing the identity. The registration
and import checks read the installed entry point, so run the suite after pip install -e ..
A publishing binding¶
Subclass PublishedDatasetContract from altar.testing with four fixtures:
source_class: the source class;built_dataset: a small dataset the binding's own builder wrote during the test session;built_release_fields: the footer fields the binding declares for that release, computed as its published dataset's are;known_variant_ids: variants the built dataset has records for.
A binding without a BigQuery table mode also overrides reads_bigquery_tables to return False.
This is the population-frequency binding's own test. Its fixture VCF is named like a gnomAD v4.1 file, which the builder holds to that version, so the test builds a neutrally named copy:
import shutil
from pathlib import Path
import pytest
from altar.testing import PublishedDatasetContract
from altar_popfreq import GnomADSource
from altar_popfreq.build_dataset import build_dataset
from altar_popfreq.source import PUBLISHED_RELEASE, release_fields
FIXTURE = Path(__file__).parent / "fixtures" / "gnomad.genomes.v4.1.sites.synthetic.vcf"
@pytest.fixture(scope="module")
def published_release_build(tmp_path_factory):
root = tmp_path_factory.mktemp("gnomad-published")
source = root / "synthetic.sites.vcf"
shutil.copyfile(FIXTURE, source)
build_dataset([source], root / "dataset", release=PUBLISHED_RELEASE)
return root / "dataset"
class TestGnomADPublishedDataset(PublishedDatasetContract):
@pytest.fixture
def source_class(self):
return GnomADSource
@pytest.fixture
def built_dataset(self, published_release_build):
return published_release_build
@pytest.fixture
def built_release_fields(self):
return release_fields(PUBLISHED_RELEASE)
@pytest.fixture
def known_variant_ids(self):
return ["chr1:10001:A:G"]
The suite copies the built dataset into fsspec's in-memory filesystem, at locations named like the real ones, and
points published_dataset() at it, so nothing reaches the network. It checks that:
- the locations are unique, the primary first, and each is read anonymously or with a connection's options;
- the
release_fieldsname the fields the builder writes and accept the built footers; - a source reading any location, an owned copy or an owned BigQuery table, through a connection or not, declares one identity;
altar pull <source> --to memory://..., and a pull from every location, read back the same values;pull_to_bigquerylabels a table that the binding's table mode then accepts, and the table mode refuses a table labelled with another release;- a binding without a table mode refuses
tableandreleasewith an error that names each.
altar-clinvar, altar-conservation, altar-opentargets-e2g and altar-popfreq run the suite.
Going further¶
Copy a published dataset from Python¶
pull_dataset and pull_to_bigquery, from altar.sources, do what altar pull does for any binding's published
dataset. Both are provisional. pull_to_bigquery is also Preview: it is tested against a fake BigQuery client only,
not against live BigQuery.
from altar.connections import GcpConnection
from altar.sources import get_annotation_source_registry, pull_dataset, pull_to_bigquery
published = get_annotation_source_registry().get("gnomad").published_dataset()
lab = GcpConnection(project="my-project")
copied = pull_dataset(published, "gs://my-bucket/gnomad-4.1.1-genomes", destination_connection=lab)
mirrored = pull_dataset(published, "/data/gnomad-4.1.1-genomes", location="hf")
loaded = pull_to_bigquery(published, "my-project.reference.gnomad_4_1_1_genomes", connection=lab)
- Both take
location, the published location to read, the primary by default. An unknown name raisesValueErrorthat lists the dataset's locations. Hugging Face limits anonymous requests; see Hugging Face rate limits before you pull withlocation="hf". pull_datasetreads the files anonymously, or withsource_connection, and writes withdestination_connectionor ambient credentials. A pull that stopped resumes, skipping files already copied.pull_to_bigqueryruns the load job throughconnection.bigquery_client(), labels the table with its release, and clusters it on the dataset'scluster_byunless its owncluster_bynames other columns (()for none).- Both take
progress, a callable that receives aPullProgress. It is called once withstage="start"after the files are listed, withfiles_totalandbytes_total, and frompull_datasetwithstage="copied"after each file.altar pullprints these on standard error. - Each returns a summary whose
to_dict()is whataltar pullprints, with the URI the files came from assource_uri.
A location, connection, destination or table never enters any identity.
Check a BigQuery copy's release¶
bigquery_release_labels(identity) returns the table labels that record a release, and
check_bigquery_release(client, table, identity) reads them with one metadata request and raises ValueError for
a table labelled with another source, release or build. A binding's table mode calls it when the run starts. Both
are provisional. Choose where data is read from describes the
labels for users.
Ready when¶
- A connection type refuses every credential in
from_options, keeps its options out ofrepr, and imports its provider SDK only inside the methods that need it. ConnectionContractpasses against the installed entry point.- A published dataset lists the primary location first, and every location holds the same builder output.
release_metadataandrelease_fieldsdescribe what the builder writes, includinggenome_build.- The source's
from_optionsresolves its options withselect_copy, and declares the same identity for every copy. PublishedDatasetContractpasses against a dataset the binding's own builder wrote.- The data's licence allows redistribution, and the source's page states where the copy is hosted and how to cite it.
Next¶
- Validate an implementation lists every conformance suite.
- Package and register a plugin covers the
altar.connectionsgroup.