Skip to content

Connections

altar.connections defines how a run reaches object storage and warehouses. A connection carries credentials and endpoints only, so it never changes the identity of a source or of stored scores. Pass credentials through connections shows the run-spec syntax.

Provisional: every name in this module is provisional. See Provisional names.

A new connection type subclasses Connection, registers under the altar.connections entry-point group, and runs altar.testing.ConnectionContract in its own tests; see Add connections and published datasets.

connections

Provisional named-connection contract: how a run reaches object storage and warehouses.

A Connection is a durable, non-secret description of how to reach data, resolved to live clients only when a consumer asks. An annotation source takes one in from_options, and FsspecStorage takes one in its constructor; a run spec declares them under [connections.<name>] and altar.run.run resolves them through the altar.connections entry-point group. Connection types implement Connection through this module.

A connection is operational: it is never part of an annotation source's, a model's, or a cache's identity.

Every name in this module is provisional and listed in __provisional__: it remains importable, and it is expected to change. altar.testing.ConnectionContract checks a connection type, and core runs it against the shipped gcp and fsspec types; the run driver, FsspecStorage, the BigQuery stores, and the gnomAD (altar-popfreq) and ClinVar (altar-clinvar) sources read through them. See Provisional names.

Connection

How to reach object storage or a warehouse, resolved to live clients only when a consumer asks.

Provisional: exported and discoverable, and expected to change (see the module docstring).

Each subclass sets name to its altar.connections registry key. A run spec builds one through from_options with the section's options; a host builds one in code and passes it to altar.run.run as connections. A connection is never part of any identity, so a consumer must not hash or record it.

from_options classmethod

from_options(**options: Any) -> Self

Build this connection from a run spec's [connections.<name>.options] mapping.

The default passes the options to the constructor. A type whose constructor could also accept a secret overrides this to refuse one, since a spec is durable.

storage_options

storage_options(for_uri: str) -> dict[str, Any]

Return the fsspec keyword arguments that open for_uri through this connection.

for_uri is a full URI or a bare scheme:// prefix, since a filesystem opened for one scheme serves every path under it. A local path needs no options, and every shipped type returns {} for one. The default raises CapabilityNotSupported: this connection does not reach object storage.

bigquery_client

bigquery_client() -> Any

Return a new google.cloud.bigquery.Client; the caller owns it and closes it.

The default raises CapabilityNotSupported: this connection does not reach BigQuery.

FsspecConnection

FsspecConnection(
    *,
    storage_options: Mapping[str, Any] | None = None,
    protocols: Sequence[str] | None = None,
)

Bases: Connection

Any fsspec filesystem (s3://, az://, https://, hf://) with fixed storage options.

Parameters:

Name Type Description Default
storage_options Mapping[str, Any] | None

fsspec keyword arguments for the filesystem, such as {"anon": True} for a public S3 bucket, {"endpoint_url": ...} for an S3-compatible store, or {"profile": "lab"} to pick a named AWS profile. They are returned unchanged for every URI this connection serves.

None
protocols Sequence[str] | None

The URI schemes this connection serves, such as ["s3"]. Asking for another scheme raises CapabilityNotSupported, so options meant for one provider never reach another. When omitted, the connection serves every non-local scheme.

None

Provider libraries find ambient credentials on their own (AWS_* variables, an Azure identity). A run spec may carry only non-secret options: from_options refuses keys such as key, secret or account_key. A host that must pass a secret builds the connection in code.

from_options classmethod

from_options(**options: Any) -> Self

Build the connection from a run spec, refusing a storage option that would carry a secret.

storage_options

storage_options(for_uri: str) -> dict[str, Any]

Return a copy of the configured options for a URI whose scheme this connection serves.

GcpConnection

GcpConnection(
    *,
    project: str | None = None,
    anonymous: bool = False,
    impersonate_service_account: str | None = None,
    requester_pays: bool = False,
)

Bases: Connection

Google Cloud Storage and BigQuery through Application Default Credentials.

Parameters:

Name Type Description Default
project str | None

The project billed for requester-pays reads and the project BigQuery runs and bills queries in. When omitted, the clients use the project the ambient credentials name.

None
anonymous bool

Read public objects with no credentials at all. BigQuery has no anonymous access, so an anonymous connection serves object storage only.

False
impersonate_service_account str | None

A service account email to act as. The ambient credentials mint short-lived tokens for it, so the caller needs roles/iam.serviceAccountTokenCreator on it.

None
requester_pays bool

Bill project for reads from requester-pays buckets; needs project.

False

No option is a secret: the credentials are whatever the environment provides (gcloud auth application-default login, a workload identity, or GOOGLE_APPLICATION_CREDENTIALS).

credentials

credentials() -> Any

Return the impersonated credentials, built once, or None to let each client find ambient ones.

An anonymous connection also returns None; its consumers ask for anonymous access explicitly.

storage_options

storage_options(for_uri: str) -> dict[str, Any]

Return gcsfs options for a gs:// URI: anonymous access, or the project and credentials to use.

bigquery_client

bigquery_client() -> Any

Return a new BigQuery client billed to project, acting as the impersonated account if one is set.

get_connection_registry

get_connection_registry() -> PluginRegistry[
    type[Connection]
]

Return the shared altar.connections registry.

Provisional, like the Connection contract it resolves; see the module docstring.