Skip to content

Add connections and published datasets

A connection type tells Altar how to reach object storage or a warehouse: which client options and credentials to use for a URI. A published dataset is a copy of a source's data that its publisher hosts, so users read it in place instead of building it. Both interfaces are Provisional: they are expected to change (see Provisional names).

When to choose a connection type or a published dataset

  • Write a connection type when your data lives in a system the shipped gcp and fsspec types don't reach, or that needs options they can't express. Most object stores need no new type: an fsspec connection passes any fsspec options. Choose where data is read from shows how a run spec uses connections.
  • Publish a dataset from a binding when its data is openly licensed and expensive to build, and you host a built copy at a public location.
  • To read through a connection from your own source, you need neither: take a connection keyword in from_options, as Read through a named connection shows.

Add a connection type

Subclass Connection, imported from altar.connections. Set name to the registry key, and override the methods the type supports:

Member Required? What it is
name required The registry name, and the type of a run spec's [connections.<name>] section.
storage_options(for_uri) default: raises CapabilityNotSupported The fsspec keyword arguments that open a URI, or a bare scheme:// prefix. Returns {} for a local path.
bigquery_client() default: raises CapabilityNotSupported A new google.cloud.bigquery.Client, which the caller owns and closes.
from_options(**options) default: passes the options to the constructor Builds the connection from a run spec's options. Override it to refuse a credential.

This type reaches a lab's S3-compatible object store at its own endpoint, with the credentials of a named profile that the provider library finds on its own. Put it in a module of your own, such as lab_connection.py:

from typing import Any

from altar.connections import Connection
from altar.models import CapabilityNotSupported


class LabObjectStoreConnection(Connection):
    """Reach the lab's S3-compatible object store at its endpoint, with the ambient credentials of a named profile."""

    name = "lab_s3"

    def __init__(self, *, endpoint_url: str, profile: str | None = None) -> None:
        self.endpoint_url = endpoint_url
        self.profile = profile

    def storage_options(self, for_uri: str) -> dict[str, Any]:
        scheme, separator, _ = for_uri.partition("://")
        if not separator or scheme in ("file", "local"):
            return {}  # a local path needs no credentials
        if scheme != "s3":
            raise CapabilityNotSupported(f"the {self.name!r} connection reaches s3:// only, not {for_uri}")
        options: dict[str, Any] = {"client_kwargs": {"endpoint_url": self.endpoint_url}}
        if self.profile is not None:
            options["profile"] = self.profile
        return options

A run spec declares it like any other connection, and a source, the run's storage or a store names it:

[connections.lab]
type = "lab_s3"
options = { endpoint_url = "https://objects.lab.example.org", profile = "research" }

The rules every connection type keeps:

  • A connection says how to reach data, never which data. It never enters a source's, a model's or a cache's identity.
  • A run spec is a file that is saved and shared, so it never carries a credential. The constructor here takes no secret, so the default from_options refuses one with a TypeError. A type whose constructor accepts a secret, for your own code to pass in, overrides from_options to refuse it, without repeating the secret in the message.
  • repr never shows an option. The base class's repr shows only the type and its name.
  • Import a provider SDK inside the method that needs it. The registry lists, resolves and builds a type without importing any provider library.
  • Return a new mapping from each storage_options call, the same for every URI of a scheme.

Publish a dataset from a binding

A binding declares its published copy by returning a PublishedDataset from its AnnotationSource.published_dataset() classmethod. Both are imported from altar.sources. A PublishedDataset holds:

  • locations: a tuple of PublishedLocation values, each a short name and a URI. The first is the primary, which a reader gets without naming a location. The rest are mirrors that hold the same files in the same layout.
  • identity: the contract, release and genome build of the data. A source that serves this release declares this identity, whichever location or copy it reads.
  • release_metadata: the ParquetReleaseMetadata the binding's builder writes into every file's footer.
  • release_fields: the footer fields, without their prefix, that every file of the release records. They must include genome_build, set to the identity's genome build.
  • cluster_by: the columns a BigQuery copy is clustered on, variant_id by default. A dataset read by something else, such as a link dataset read by interval, names its own.
from altar.sources import (
    AnnotationColumn,
    AnnotationContract,
    ParquetReleaseMetadata,
    PublishedDataset,
    PublishedLocation,
)

CONSERVATION = AnnotationContract(
    source_id="org.example.annotation.conservation",
    schema_version="1.0.0",
    columns=(AnnotationColumn("conservation_score", "float", "Conservation score"),),
)
RELEASE_METADATA = ParquetReleaseMetadata(
    prefix="altar_example_conservation.",
    distribution="altar-example-conservation",
    label="Example conservation",
    command="altar-example-conservation-build",
)
PUBLISHED = PublishedDataset(
    locations=(
        PublishedLocation("gcs", "gs://publisher-bucket/conservation-2026-10"),
        PublishedLocation("hf", "hf://datasets/publisher/conservation-2026-10"),
    ),
    identity=CONSERVATION.identity(release="2026-10", genome_build="hg38"),
    release_metadata=RELEASE_METADATA,
    release_fields={"release": "2026-10", "genome_build": "hg38"},
)

The source returns it, and its from_options resolves the run spec's options with PUBLISHED.select_copy(...):

from typing import Self

from altar.sources import AnnotationSource, PublishedCopy


class ConservationSource(AnnotationSource):
    name = "example_conservation"

    @classmethod
    def published_dataset(cls) -> PublishedDataset:
        return PUBLISHED

    @classmethod
    def from_options(cls, *, dataset=None, location=None, connection=None) -> Self:
        copy = PUBLISHED.select_copy("Example conservation", dataset=dataset, location=location, connection=connection)
        uri = copy.uri if isinstance(copy, PublishedCopy) else copy.dataset
        return cls(uri, storage_options=copy.storage_options)

The rest of the class is an ordinary source over the binding's Parquet layout; see Add an annotation source. select_copy applies the same option rules and messages for every publishing binding, and opens nothing:

  • With neither dataset nor table, it returns a PublishedCopy: the location location names, the primary by default, read anonymously ({"token": "anon"} on Cloud Storage) or through the connection.
  • With dataset, it returns an OwnedDatasetCopy: the user's Parquet copy, read through the connection or with the provider's ambient credentials.
  • With table, it returns an OwnedTableCopy: the user's BigQuery table, which needs a connection. release names the release the table holds, by default the published one.
  • It raises ValueError for options that name no single copy, such as both dataset and table, or location with either.

A binding without a BigQuery table mode takes neither table nor release in from_options, and refuses each by name.

Each location holds the builder's output, the same files at every location: Parquet files under <partition column>=<chromosome>/ directories, each holding the partition column and variant_id, and recording the release in its footer. altar pull refuses a published dataset without release_metadata, because it could not check a copy. List first the location the binding's base dependencies can read. A location's name is short and lowercase (letters, digits, _ and -, starting with a letter), and is what users type after --from. Name a Cloud Storage location by its gs:// URI: a partitioned dataset is found by listing its directory, which plain HTTPS cannot do there.

A binding publishes one release, its latest, and a binding version pins the release it reads by default. Publish a dataset release is the maintainers' procedure for building, hosting and bumping a release.

Register the entry point

Register a connection type under the altar.connections group, with its name as the key:

[project.entry-points."altar.connections"]
lab_s3 = "lab_connection:LabObjectStoreConnection"

A publishing binding registers its source under altar.annotation_sources as usual. altar sources, altar pull and the run driver find its published dataset through the registry.

Test with the conformance suites

A connection type

Subclass ConnectionContract from altar.testing, and describe the type with its fixtures:

import pytest
from altar.testing import ConnectionContract

from lab_connection import LabObjectStoreConnection


class TestLabObjectStoreConnection(ConnectionContract):
    @pytest.fixture
    def connection_type(self):
        return LabObjectStoreConnection

    @pytest.fixture
    def connection_options(self):
        return {"endpoint_url": "https://objects.lab.example.org", "profile": "research"}

    @pytest.fixture
    def served_uri(self):
        return "s3://lab-bucket/dataset"

The other fixtures are optional: unsupported_uri (a URI whose scheme it must decline), reaches_bigquery, credential_options (spec options that carry a credential) and connection_holding_secret (a connection built in code with a secret, and the secret). Override connection to install fakes for a provider SDK first.

The suite checks that the type is registered under its name; that a local path needs no options; that a served URI gets a fresh mapping, the same for its bare scheme:// prefix; that an unserved scheme, and BigQuery when the type does not reach it, raise CapabilityNotSupported; that from_options refuses a credential without repeating it; that repr shows no option; that listing, resolving and building the type imports no provider library; and that PublishedDataset.select_copy reaches each copy through it without changing the identity. The registration and import checks read the installed entry point, so run the suite after pip install -e ..

A publishing binding

Subclass PublishedDatasetContract from altar.testing with four fixtures:

  • source_class: the source class;
  • built_dataset: a small dataset the binding's own builder wrote during the test session;
  • built_release_fields: the footer fields the binding declares for that release, computed as its published dataset's are;
  • known_variant_ids: variants the built dataset has records for.

A binding without a BigQuery table mode also overrides reads_bigquery_tables to return False.

This is the population-frequency binding's own test. Its fixture VCF is named like a gnomAD v4.1 file, which the builder holds to that version, so the test builds a neutrally named copy:

import shutil
from pathlib import Path

import pytest
from altar.testing import PublishedDatasetContract
from altar_popfreq import GnomADSource
from altar_popfreq.build_dataset import build_dataset
from altar_popfreq.source import PUBLISHED_RELEASE, release_fields

FIXTURE = Path(__file__).parent / "fixtures" / "gnomad.genomes.v4.1.sites.synthetic.vcf"


@pytest.fixture(scope="module")
def published_release_build(tmp_path_factory):
    root = tmp_path_factory.mktemp("gnomad-published")
    source = root / "synthetic.sites.vcf"
    shutil.copyfile(FIXTURE, source)
    build_dataset([source], root / "dataset", release=PUBLISHED_RELEASE)
    return root / "dataset"


class TestGnomADPublishedDataset(PublishedDatasetContract):
    @pytest.fixture
    def source_class(self):
        return GnomADSource

    @pytest.fixture
    def built_dataset(self, published_release_build):
        return published_release_build

    @pytest.fixture
    def built_release_fields(self):
        return release_fields(PUBLISHED_RELEASE)

    @pytest.fixture
    def known_variant_ids(self):
        return ["chr1:10001:A:G"]

The suite copies the built dataset into fsspec's in-memory filesystem, at locations named like the real ones, and points published_dataset() at it, so nothing reaches the network. It checks that:

  • the locations are unique, the primary first, and each is read anonymously or with a connection's options;
  • the release_fields name the fields the builder writes and accept the built footers;
  • a source reading any location, an owned copy or an owned BigQuery table, through a connection or not, declares one identity;
  • altar pull <source> --to memory://..., and a pull from every location, read back the same values;
  • pull_to_bigquery labels a table that the binding's table mode then accepts, and the table mode refuses a table labelled with another release;
  • a binding without a table mode refuses table and release with an error that names each.

altar-clinvar, altar-conservation, altar-opentargets-e2g and altar-popfreq run the suite.

Going further

Copy a published dataset from Python

pull_dataset and pull_to_bigquery, from altar.sources, do what altar pull does for any binding's published dataset. Both are provisional. pull_to_bigquery is also Preview: it is tested against a fake BigQuery client only, not against live BigQuery.

from altar.connections import GcpConnection
from altar.sources import get_annotation_source_registry, pull_dataset, pull_to_bigquery

published = get_annotation_source_registry().get("gnomad").published_dataset()
lab = GcpConnection(project="my-project")

copied = pull_dataset(published, "gs://my-bucket/gnomad-4.1.1-genomes", destination_connection=lab)
mirrored = pull_dataset(published, "/data/gnomad-4.1.1-genomes", location="hf")
loaded = pull_to_bigquery(published, "my-project.reference.gnomad_4_1_1_genomes", connection=lab)
  • Both take location, the published location to read, the primary by default. An unknown name raises ValueError that lists the dataset's locations. Hugging Face limits anonymous requests; see Hugging Face rate limits before you pull with location="hf".
  • pull_dataset reads the files anonymously, or with source_connection, and writes with destination_connection or ambient credentials. A pull that stopped resumes, skipping files already copied.
  • pull_to_bigquery runs the load job through connection.bigquery_client(), labels the table with its release, and clusters it on the dataset's cluster_by unless its own cluster_by names other columns (() for none).
  • Both take progress, a callable that receives a PullProgress. It is called once with stage="start" after the files are listed, with files_total and bytes_total, and from pull_dataset with stage="copied" after each file. altar pull prints these on standard error.
  • Each returns a summary whose to_dict() is what altar pull prints, with the URI the files came from as source_uri.

A location, connection, destination or table never enters any identity.

Check a BigQuery copy's release

bigquery_release_labels(identity) returns the table labels that record a release, and check_bigquery_release(client, table, identity) reads them with one metadata request and raises ValueError for a table labelled with another source, release or build. A binding's table mode calls it when the run starts. Both are provisional. Choose where data is read from describes the labels for users.

Ready when

  • A connection type refuses every credential in from_options, keeps its options out of repr, and imports its provider SDK only inside the methods that need it.
  • ConnectionContract passes against the installed entry point.
  • A published dataset lists the primary location first, and every location holds the same builder output.
  • release_metadata and release_fields describe what the builder writes, including genome_build.
  • The source's from_options resolves its options with select_copy, and declares the same identity for every copy.
  • PublishedDatasetContract passes against a dataset the binding's own builder wrote.
  • The data's licence allows redistribution, and the source's page states where the copy is hosted and how to cite it.

Next