Connections¶
altar.connections defines how a run reaches object storage and warehouses. A connection carries credentials and
endpoints only, so it never changes the identity of a source or of stored scores.
Pass credentials through connections shows the
run-spec syntax.
Provisional: every name in this module is provisional. See Provisional names.
A new connection type subclasses Connection, registers under the altar.connections entry-point group, and runs
altar.testing.ConnectionContract in its own tests; see
Add connections and published datasets.
connections
¶
Provisional named-connection contract: how a run reaches object storage and warehouses.
A Connection is a durable, non-secret description of how to reach data, resolved to live clients only when a
consumer asks. An annotation source takes one in from_options, and FsspecStorage takes one in its
constructor; a run spec declares them under [connections.<name>] and altar.run.run resolves them through
the altar.connections entry-point group. Connection types implement Connection through this module.
A connection is operational: it is never part of an annotation source's, a model's, or a cache's identity.
Every name in this module is provisional and listed in __provisional__: it remains importable, and it is
expected to change. altar.testing.ConnectionContract checks a connection type, and core runs
it against the shipped gcp and fsspec types; the run driver, FsspecStorage, the BigQuery stores, and the
gnomAD (altar-popfreq) and ClinVar (altar-clinvar) sources read through them. See
Provisional names.
Connection
¶
How to reach object storage or a warehouse, resolved to live clients only when a consumer asks.
Provisional: exported and discoverable, and expected to change (see the module docstring).
Each subclass sets name to its altar.connections registry key. A run spec builds one through
from_options with the section's options; a host builds one in code and passes it to altar.run.run
as connections. A connection is never part of any identity, so a consumer must not hash or record it.
from_options
classmethod
¶
Build this connection from a run spec's [connections.<name>.options] mapping.
The default passes the options to the constructor. A type whose constructor could also accept a secret overrides this to refuse one, since a spec is durable.
storage_options
¶
Return the fsspec keyword arguments that open for_uri through this connection.
for_uri is a full URI or a bare scheme:// prefix, since a filesystem opened for one scheme
serves every path under it. A local path needs no options, and every shipped type returns {} for one.
The default raises CapabilityNotSupported: this connection does not reach object storage.
bigquery_client
¶
Return a new google.cloud.bigquery.Client; the caller owns it and closes it.
The default raises CapabilityNotSupported: this connection does not reach BigQuery.
FsspecConnection
¶
FsspecConnection(
*,
storage_options: Mapping[str, Any] | None = None,
protocols: Sequence[str] | None = None,
)
Bases: Connection
Any fsspec filesystem (s3://, az://, https://, hf://) with fixed storage options.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
storage_options
|
Mapping[str, Any] | None
|
fsspec keyword arguments for the filesystem, such as |
None
|
protocols
|
Sequence[str] | None
|
The URI schemes this connection serves, such as |
None
|
Provider libraries find ambient credentials on their own (AWS_* variables, an Azure identity). A run
spec may carry only non-secret options: from_options refuses keys such as key, secret or
account_key. A host that must pass a secret builds the connection in code.
GcpConnection
¶
GcpConnection(
*,
project: str | None = None,
anonymous: bool = False,
impersonate_service_account: str | None = None,
requester_pays: bool = False,
)
Bases: Connection
Google Cloud Storage and BigQuery through Application Default Credentials.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
project
|
str | None
|
The project billed for requester-pays reads and the project BigQuery runs and bills queries in. When omitted, the clients use the project the ambient credentials name. |
None
|
anonymous
|
bool
|
Read public objects with no credentials at all. BigQuery has no anonymous access, so an anonymous connection serves object storage only. |
False
|
impersonate_service_account
|
str | None
|
A service account email to act as. The ambient credentials mint
short-lived tokens for it, so the caller needs |
None
|
requester_pays
|
bool
|
Bill |
False
|
No option is a secret: the credentials are whatever the environment provides (gcloud auth
application-default login, a workload identity, or GOOGLE_APPLICATION_CREDENTIALS).
credentials
¶
Return the impersonated credentials, built once, or None to let each client find ambient ones.
An anonymous connection also returns None; its consumers ask for anonymous access explicitly.
storage_options
¶
Return gcsfs options for a gs:// URI: anonymous access, or the project and credentials to use.
bigquery_client
¶
Return a new BigQuery client billed to project, acting as the impersonated account if one is set.
get_connection_registry
¶
get_connection_registry() -> PluginRegistry[
type[Connection]
]
Return the shared altar.connections registry.
Provisional, like the Connection contract it resolves; see the module docstring.