Skip to content

Choose where data is read from

The gnomAD, ClinVar, Open Targets E2G and phyloP sources read data that the Kundaje lab builds from each upstream release and hosts publicly. By default a source reads that copy where it is published, anonymously. A source can read the same release from the Hugging Face mirror, from your own copy or from a BigQuery table instead. altar pull makes a copy, and a connection passes credentials to a source. altar pull and [connections] are Provisional. Reading a BigQuery table and loading one with altar pull --bigquery are Preview.

A source reads one of four copies of its release:

Copy What it is How the source section names it
Published copy Files in a public Google Cloud bucket, gs://kundajelab-altar nothing; it is the default
Mirror The same files in a public Hugging Face dataset location = "hf"
Your copy The files on your disk or in your bucket, copied with altar pull --to or built with the binding's builder dataset = "<path or URI>"
Your BigQuery table The files loaded into your table with altar pull --bigquery table = "<project.dataset.table>" and a gcp connection

Every copy holds the same release, so the source fills the same columns and records the same release and build whichever copy it reads.

Before you start

  • The binding of each source, such as pip install altar-popfreq. Every binding that publishes data installs altar[gcs], which reads gs:// locations.
  • For the Hugging Face mirror: pip install "altar[hf]".
  • For BigQuery: pip install "altar[bigquery]", a Google Cloud project, and Application Default Credentials, from gcloud auth application-default login, a workload identity or GOOGLE_APPLICATION_CREDENTIALS.
  • For a copy: disk or bucket space. gnomAD 4.1.1 genomes is 24 files and 24.9 GiB.

The examples use the quickstart's annotate.toml.

Read the published copy

A source with no dataset or table reads the published copy, anonymously, with no download. A lookup reads only the parts of the dataset its variants fall in. The run says where each source reads:

[sources.gnomad] opened gnomad 4.1.1-genomes (hg38) from gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes (published, gcs) in 2.6s

altar sources <name> lists a source's published locations and release:

$ altar sources gnomad
...
Data:
  Published: release 4.1.1-genomes on hg38, read where it is published unless you name your own copy.
    gcs: gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes (primary)
    hf:  hf://datasets/altar-genomics/gnomad-4.1.1-genomes
  Copy it: altar pull gnomad --to PATH_OR_URI  (or --bigquery PROJECT.DATASET.TABLE)
  Or build your own copy with: altar-popfreq-build --help
  Genome builds: hg38

Each source's page lists its published locations under "Where the data comes from", such as gnomAD.

location picks the copy a source reads. gcs, the bucket, is the default. hf reads the Hugging Face mirror, which holds the same files:

[sources.gnomad]
type = "gnomad"
options = { location = "hf" }

Pick the mirror when it is closer or faster, or reachable where the bucket is not.

Hugging Face rate limits

Hugging Face limits how many requests an anonymous client makes in a few minutes. A lookup reads one stretch of a file for each cluster of nearby variants, so a large, scattered cohort against a dense dataset can reach the limit and wait. 10,000 scattered SNVs against phyloP 100-way took 44 s from gcs and 126 to 563 s from hf. For a large cohort, read gcs, or copy the dataset once with altar pull and read your copy.

Read your own copy

dataset names your copy of the files, as a local path or a URI:

[sources.gnomad]
type = "gnomad"
options = { dataset = "/data/gnomad-4.1.1-genomes" }

Your copy is one that altar pull --to made, or one the binding's builder wrote. Each binding pins the release it reads by default, and publishes only that one. To read another release, such as a newer ClinVar or ClinVar on hg19, build it with the builder and name it with dataset; each source's page says how, such as ClinVar. The source reads the release and genome build from the copy's files, and the run checks the build against [run] genome_build.

A copy in a private bucket needs credentials; see Pass credentials through connections.

Read from BigQuery

Preview

Reading a source from BigQuery is Preview. It is tested on the BigQuery emulator and DuckDB, and has not been run against live BigQuery.

table names a BigQuery table that holds the source's columns, such as one altar pull --bigquery loaded. The source queries it with a gcp connection's credentials:

[connections.lab]
type = "gcp"
options = { project = "my-project" }

[sources.gnomad]
type = "gnomad"
options = { table = "my-project.reference.gnomad_4_1_1_genomes" }
connection = "lab"

Queries run in, and are billed to, the connection's project. When the run starts, the source reads the release labels altar pull wrote on the table, with one metadata request. It refuses a table labelled with another release or build, and warns about a table with no labels, such as one loaded by hand. release names the release such a table holds when it is not the published one.

gnomAD, ClinVar and Open Targets E2G take table. phyloP reads files only. Keep scores in BigQuery keeps a run's scores in the same project, and Run at warehouse scale on BigQuery joins these sources to the scores in SQL from Python.

Copy a dataset with altar pull

altar pull copies a source's published files to a path or a bucket, or loads them into a BigQuery table:

altar pull gnomad --to /data/gnomad-4.1.1-genomes
altar pull gnomad --from hf --to /data/gnomad-4.1.1-genomes
altar pull gnomad --bigquery my-project.reference.gnomad_4_1_1_genomes --project my-project

--to copies the Parquet files, keeping their partition directories. --from hf reads them from the mirror. The command prints the size before it starts, then a line per file with an estimate of the time left:

$ altar pull gnomad --to /data/gnomad-4.1.1-genomes
copying gnomad 4.1.1-genomes (hg38) from gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes to /data/gnomad-4.1.1-genomes: 24 files, 26.7 GB
[1/24] chr=chr1/gnomad.genomes.v4.1.1.sites.chr1.parquet (2.0 GB); 2.0 GB of 26.7 GB, 1m05s elapsed, about 13m02s left
...
copied 24 files (26.7 GB) in 14m02s; every file records gnomad 4.1.1-genomes (hg38)

If the copy stops partway, run the same command again. It skips the files already there:

resuming: 9 of 24 files (12.2 GB) are already there and are skipped; 15 to copy (14.5 GB)

The command checks that every file it copied or skipped records the published release and build. It refuses a destination that holds files that are not the dataset's, unless you pass --overwrite.

--bigquery loads the published files into a table with one BigQuery load job, and labels the table with the release. It is Preview: tested against a fake BigQuery client only. The BigQuery dataset must exist already. The published bucket is in the US multi-region, so it loads directly only into a dataset in the US location:

bq mk --dataset --location=US my-project:reference

For a dataset in another region, copy the files to a bucket in that region with --to, then load that copy with --bigquery ... --source-uri gs://<your copy>. --project names the project the load runs in and is billed to.

The command-line reference lists every flag, including --impersonate-service-account and --source-project for a requester-pays bucket.

Pass credentials through connections

A [connections.<name>] section declares one way of reaching a bucket or a warehouse: its credentials and project. A source, [storage], [store] or [detail_store] uses it by naming it with connection; Choose where results are stored shows the last three. This spec reads your gnomAD copy from your bucket, billed to your project:

[connections.lab]
type = "gcp"
options = { project = "my-project" }

[sources.gnomad]
type = "gnomad"
options = { dataset = "gs://my-bucket/gnomad-4.1.1-genomes" }
connection = "lab"

There are two connection types:

  • gcp reaches Cloud Storage and BigQuery through Application Default Credentials. It can name a project, a service account to impersonate, or anonymous = true for public objects.
  • fsspec reaches any other fsspec filesystem, such as s3:// or az://, with the provider's own ambient credentials.

A connection says how to reach data, never which data a source reads, so a source's release and stored results are the same through any connection. A spec never holds a secret: an fsspec connection refuses keys such as key, secret and token. A program that holds a secret builds the connection in Python and passes it to the run; see Use Altar from Python. The run spec reference lists each type's options.

Without a connection, the published copy is read anonymously and your copy with your environment's default credentials.

Check the result

The run's line for each source names the copy it opened:

  • the published copy: from gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes (published, gcs), or (published, hf) from the mirror;
  • your copy: from /data/gnomad-4.1.1-genomes;
  • a table: from BigQuery table my-project.reference.gnomad_4_1_1_genomes.

The run summary's sources records the release and build each source served, the same whichever copy it read.

If it fails

  • A missing extra stops the run and names it:

    altar run: error: [sources.phylop]: reading hf:// needs the hf extra: pip install "altar[hf]"
    
  • altar pull stops with exit status 2 and one line when Application Default Credentials are missing or expired, the BigQuery dataset does not exist, or your credentials lack a permission. The line says what to do next, such as gcloud auth application-default login or bq mk --dataset.

Fix a run that stops or warns covers the other messages.

Next