Choose where data is read from¶
The gnomAD, ClinVar, Open Targets E2G and phyloP sources read data that the Kundaje lab builds from each upstream
release and hosts publicly. By default a source reads that copy where it is published, anonymously. A source can
read the same release from the Hugging Face mirror, from your own copy or from a BigQuery table instead.
altar pull makes a copy, and a connection passes credentials to a source. altar pull and [connections] are
Provisional. Reading a BigQuery table and loading one with
altar pull --bigquery are Preview.
A source reads one of four copies of its release:
| Copy | What it is | How the source section names it |
|---|---|---|
| Published copy | Files in a public Google Cloud bucket, gs://kundajelab-altar |
nothing; it is the default |
| Mirror | The same files in a public Hugging Face dataset | location = "hf" |
| Your copy | The files on your disk or in your bucket, copied with altar pull --to or built with the binding's builder |
dataset = "<path or URI>" |
| Your BigQuery table | The files loaded into your table with altar pull --bigquery |
table = "<project.dataset.table>" and a gcp connection |
Every copy holds the same release, so the source fills the same columns and records the same release and build whichever copy it reads.
Before you start¶
- The binding of each source, such as
pip install altar-popfreq. Every binding that publishes data installsaltar[gcs], which readsgs://locations. - For the Hugging Face mirror:
pip install "altar[hf]". - For BigQuery:
pip install "altar[bigquery]", a Google Cloud project, and Application Default Credentials, fromgcloud auth application-default login, a workload identity orGOOGLE_APPLICATION_CREDENTIALS. - For a copy: disk or bucket space. gnomAD 4.1.1 genomes is 24 files and 24.9 GiB.
The examples use the quickstart's annotate.toml.
Read the published copy¶
A source with no dataset or table reads the published copy, anonymously, with no download. A lookup reads only
the parts of the dataset its variants fall in. The run says where each source reads:
[sources.gnomad] opened gnomad 4.1.1-genomes (hg38) from gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes (published, gcs) in 2.6s
altar sources <name> lists a source's published locations and release:
$ altar sources gnomad
...
Data:
Published: release 4.1.1-genomes on hg38, read where it is published unless you name your own copy.
gcs: gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes (primary)
hf: hf://datasets/altar-genomics/gnomad-4.1.1-genomes
Copy it: altar pull gnomad --to PATH_OR_URI (or --bigquery PROJECT.DATASET.TABLE)
Or build your own copy with: altar-popfreq-build --help
Genome builds: hg38
Each source's page lists its published locations under "Where the data comes from", such as gnomAD.
location picks the copy a source reads. gcs, the bucket, is the default. hf reads the Hugging Face mirror,
which holds the same files:
Pick the mirror when it is closer or faster, or reachable where the bucket is not.
Hugging Face rate limits¶
Hugging Face limits how many requests an anonymous client makes in a few minutes. A lookup reads one stretch of a
file for each cluster of nearby variants, so a large, scattered cohort against a dense dataset can reach the limit
and wait. 10,000 scattered SNVs against phyloP 100-way took 44 s from gcs and 126 to 563 s from hf. For a large
cohort, read gcs, or copy the dataset once with altar pull and read your copy.
Read your own copy¶
dataset names your copy of the files, as a local path or a URI:
Your copy is one that altar pull --to made, or one the binding's builder wrote. Each binding pins the release it
reads by default, and publishes only that one. To read another release, such as a newer ClinVar or ClinVar on
hg19, build it with the builder and name it with dataset; each source's page says how, such as
ClinVar. The source reads the release and genome build from
the copy's files, and the run checks the build against [run] genome_build.
A copy in a private bucket needs credentials; see Pass credentials through connections.
Read from BigQuery¶
Preview
Reading a source from BigQuery is Preview. It is tested on the BigQuery emulator and DuckDB, and has not been run against live BigQuery.
table names a BigQuery table that holds the source's columns, such as one altar pull --bigquery loaded. The
source queries it with a gcp connection's credentials:
[connections.lab]
type = "gcp"
options = { project = "my-project" }
[sources.gnomad]
type = "gnomad"
options = { table = "my-project.reference.gnomad_4_1_1_genomes" }
connection = "lab"
Queries run in, and are billed to, the connection's project. When the run starts, the source reads the release
labels altar pull wrote on the table, with one metadata request. It refuses a table labelled with another
release or build, and warns about a table with no labels, such as one loaded by hand. release names the release
such a table holds when it is not the published one.
gnomAD, ClinVar and Open Targets E2G take table. phyloP reads files only.
Keep scores in BigQuery keeps a run's scores in the same project, and
Run at warehouse scale on BigQuery joins these sources to the scores in SQL from Python.
Copy a dataset with altar pull¶
altar pull copies a source's published files to a path or a bucket, or loads them into a BigQuery table:
altar pull gnomad --to /data/gnomad-4.1.1-genomes
altar pull gnomad --from hf --to /data/gnomad-4.1.1-genomes
altar pull gnomad --bigquery my-project.reference.gnomad_4_1_1_genomes --project my-project
--to copies the Parquet files, keeping their partition directories. --from hf reads them from the mirror. The
command prints the size before it starts, then a line per file with an estimate of the time left:
$ altar pull gnomad --to /data/gnomad-4.1.1-genomes
copying gnomad 4.1.1-genomes (hg38) from gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes to /data/gnomad-4.1.1-genomes: 24 files, 26.7 GB
[1/24] chr=chr1/gnomad.genomes.v4.1.1.sites.chr1.parquet (2.0 GB); 2.0 GB of 26.7 GB, 1m05s elapsed, about 13m02s left
...
copied 24 files (26.7 GB) in 14m02s; every file records gnomad 4.1.1-genomes (hg38)
If the copy stops partway, run the same command again. It skips the files already there:
The command checks that every file it copied or skipped records the published release and build. It refuses a
destination that holds files that are not the dataset's, unless you pass --overwrite.
--bigquery loads the published files into a table with one BigQuery load job, and labels the table with the
release. It is Preview: tested against a fake BigQuery client only. The BigQuery dataset must exist already.
The published bucket is in the US multi-region, so it loads directly only into a dataset in the US location:
For a dataset in another region, copy the files to a bucket in that region with --to, then load that copy with
--bigquery ... --source-uri gs://<your copy>. --project names the project the load runs in and is billed to.
The command-line reference lists every flag, including
--impersonate-service-account and --source-project for a requester-pays bucket.
Pass credentials through connections¶
A [connections.<name>] section declares one way of reaching a bucket or a warehouse: its credentials and
project. A source, [storage], [store] or [detail_store] uses it by naming it with connection;
Choose where results are stored shows the last three. This spec reads your gnomAD copy from your
bucket, billed to your project:
[connections.lab]
type = "gcp"
options = { project = "my-project" }
[sources.gnomad]
type = "gnomad"
options = { dataset = "gs://my-bucket/gnomad-4.1.1-genomes" }
connection = "lab"
There are two connection types:
gcpreaches Cloud Storage and BigQuery through Application Default Credentials. It can name aproject, a service account to impersonate, oranonymous = truefor public objects.fsspecreaches any other fsspec filesystem, such ass3://oraz://, with the provider's own ambient credentials.
A connection says how to reach data, never which data a source reads, so a source's release and stored results are
the same through any connection. A spec never holds a secret: an fsspec connection refuses keys such as key,
secret and token. A program that holds a secret builds the connection in Python and passes it to the run;
see Use Altar from Python. The run spec reference lists
each type's options.
Without a connection, the published copy is read anonymously and your copy with your environment's default credentials.
Check the result¶
The run's line for each source names the copy it opened:
- the published copy:
from gs://kundajelab-altar/popfreq/gnomad-4.1.1-genomes (published, gcs), or(published, hf)from the mirror; - your copy:
from /data/gnomad-4.1.1-genomes; - a table:
from BigQuery table my-project.reference.gnomad_4_1_1_genomes.
The run summary's sources records the release and build each source served, the same whichever copy it read.
If it fails¶
-
A missing extra stops the run and names it:
-
altar pullstops with exit status 2 and one line when Application Default Credentials are missing or expired, the BigQuery dataset does not exist, or your credentials lack a permission. The line says what to do next, such asgcloud auth application-default loginorbq mk --dataset.
Fix a run that stops or warns covers the other messages.
Next¶
- Score variants with a model adds a model's predictions beside the annotations.
- Choose where results are stored keeps scores in BigQuery beside these tables.
- Add connections and published datasets adds a connection type, or publishes a binding's own dataset.