Publish a dataset release¶
To publish a binding's derived dataset, build a new upstream release into the binding's Parquet layout, host it, and point the binding at it. Users read the published copy as described in Choose where data is read from.
Four bindings publish a dataset:
| Binding | Source | Current release and locations |
|---|---|---|
altar-popfreq |
gnomad |
gnomAD |
altar-clinvar |
clinvar |
ClinVar |
altar-opentargets-e2g |
opentargets_e2g |
Open Targets E2G |
altar-conservation |
phylop100way |
phyloP 100-way |
Each dataset is published to the project's public bucket, gs://kundajelab-altar, and mirrored as a public dataset
in the altar-genomics Hugging Face organization.
Rules¶
- A release is published once, at its own prefix:
gs://kundajelab-altar/<binding directory>/<dataset name>/. The dataset name says what the data is and includes every part that tells one published copy from another: the release, the dataset's variant set, and the genome build when the data exists on more than one. The four today arepopfreq/gnomad-4.1.1-genomes,clinvar/clinvar-20260928-hg38,opentargets-e2g/enhancer-to-gene-26.09andconservation/phylop100way-hg38. phyloP's name has no release because UCSC publishes one phyloP 100-way track per genome build. - The mirror is the Hugging Face dataset
altar-genomics/<dataset name>. The organization is flat, so a dataset name that needs its bucket directory to say what it is takes the binding directory instead:opentargets-e2g/enhancer-to-gene-26.09is mirrored asaltar-genomics/opentargets-e2g-26.09. - A new release keeps its dataset's pattern. ClinVar 20261005 goes to
clinvar/clinvar-20261005-hg38/andaltar-genomics/clinvar-20261005-hg38; Open Targets 26.12 goes toopentargets-e2g/enhancer-to-gene-26.12/andaltar-genomics/opentargets-e2g-26.12. - A published prefix is immutable. Never overwrite or delete a prefix that a binding version released or merged
to
mainpoints at. A binding version reads its pinned prefix by default, so old installs keep reading it after a bump. - Never run a builder with
--overwriteat a published prefix.--overwritereplaces the previous build, so readers would find files missing or changed. - A layout-only rebuild may replace files in place. It rebuilds the same release with the same content and the same file names, and changes only the physical layout, such as the row-group size. The identity is unchanged, so the prefix keeps its name. See Replace a release's layout.
Bump to a new release¶
1. Pick the upstream release¶
Use an upstream release whose files are final and dated. See the per-dataset notes for where each source publishes its releases.
2. Build to the new prefix¶
Build on a server near the data, or on a short-lived cloud VM for a large source. A laptop connection is too slow
and too easily interrupted. The builders stream --input URIs and write --output URIs
directly, so no download or copy step is needed:
altar-clinvar-build --release 20261005 --genome-build hg38 \
--input https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar_20261005.vcf.gz \
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar_20261005_papu.vcf.gz \
--output gs://kundajelab-altar/clinvar/clinvar-20261005-hg38/
The ClinVar and gnomAD builders write to a bucket URI directly, as above. The Open Targets E2G and phyloP builders write a local directory: build there, then upload the directory to the new prefix, as their per-dataset notes show.
The writing account needs object write access to the bucket for the length of the build (see Access).
Each finished file is committed whole, so a reader never sees a partial file. If the build stops, run the same
command again: it resumes from the inputs recorded under _altar_build/. The build is complete when the command
prints its summary and _altar_build/ is gone from the prefix:
The command must report that no objects match. Keep the summary line, which gives the alleles written, the VCF records read, and the records skipped by reason. A resumed run counts only the inputs it built itself.
3. Point the binding at it¶
On a branch, edit the module that pins the binding's published release:
| Binding | Module | What to change |
|---|---|---|
altar-popfreq |
altar_popfreq/source.py |
PUBLISHED_RELEASE, PUBLISHED_DATASET's gcs and hf URIs, and PUBLISHED_RELEASE's docstring, which names the upstream files |
altar-clinvar |
altar_clinvar/source.py |
the same: PUBLISHED_RELEASE, such as ClinVarRelease(release_date="20261005", genome_build="hg38"), PUBLISHED_DATASET and the docstring |
altar-opentargets-e2g |
altar_opentargets_e2g/annotation.py |
PUBLISHED_RELEASE, such as "26.12", PUBLISHED_DATASET's URIs, and PUBLISHED_RELEASE's docstring, which names the release date |
altar-conservation |
altar_conservation/tracks.py |
the track's ConservationTrack: release, upstream_url, upstream_md5, and the path its published copy is named by |
Install the branch, for example with uvx uv@0.12.17 sync --all-packages in the repository.
4. Verify¶
Copy the new prefix to scratch space with the branch's altar pull. It checks that every file records the
release and genome build that the published dataset now declares:
Then check the copy:
-
Count its rows and compare with the builder's summary:
-
Compare with upstream. The VCF records the builder read are the upstream record count. Alleles written plus records skipped must equal it. See the per-dataset notes for how to count upstream.
- Compare the file count, the total size, and the rows per chromosome with the previous release. A release grows slowly; a chromosome that shrank or vanished is a failed input.
- Spot-check a few variants you know, such as in BRCA1 or CFTR, in the old and the new release, and check that each is present with a plausible value.
5. Mirror to Hugging Face¶
Mirror the verified copy, so the mirror holds the same files byte for byte. Write the dataset card first, as
README.md in the copy's directory. The Open Targets E2G and phyloP cards are kept in the repository, in
bindings/opentargets-e2g/hf-card/README.md and bindings/conservation/hf-card/README.md; update that file and
copy it. For gnomAD and ClinVar, start from the previous release's card:
hf download altar-genomics/clinvar-20260928-hg38 README.md --type dataset --local-dir /scratch/card
cp /scratch/card/README.md /scratch/clinvar-20261005-hg38/README.md
Update every release-specific value in it: pretty_name, size_categories, the title, the upstream file names,
the row count, file count and size, the records skipped by reason, the footer values, the gs:// URI, and the
paths in the DuckDB example. Keep its other sections: what the data is derived from and that it is a derived
copy, how it was made (builder and builder version), the layout and columns, how to read it with and without
Altar, and the upstream terms and citation.
Create the repository private, upload, check the card and the dataset viewer, then make it public:
hf repos create altar-genomics/clinvar-20261005-hg38 --type dataset --private
hf upload altar-genomics/clinvar-20261005-hg38 /scratch/clinvar-20261005-hg38 . --type dataset
hf repos settings altar-genomics/clinvar-20261005-hg38 --type dataset --public
hf upload makes one commit. For a large dataset, such as gnomAD, hf upload-large-folder uploads in resumable
chunks instead. Read the mirror once with altar pull clinvar --from hf --to <another scratch directory>, which
needs altar[hf]. For gnomAD, a run spec with location = "hf" that looks up a few variants is enough.
6. Open one pull request¶
One pull request moves the binding to the new release. It holds:
- the changes from step 3;
- the tests that assert the published URIs and release:
tests/test_source.pyforaltar-popfreqandaltar-clinvar,tests/test_link_dataset.pyforaltar-opentargets-e2g, andtests/test_conservation.pyforaltar-conservation; - the release, URIs and Hugging Face link in the binding's README, and its
hf-card/README.mdwhere it has one; - every page that names the release as the published one, such as the source's page under
Models and data.
git grepthe old release to find them. The builder tests use their own fixture dates and stay as they are.
Run the gate, merge, and release the binding as usual. Do not delete the old prefix or the old Hugging Face dataset.
What users see¶
A new release is a new source identity: the contract, the new release label, and the genome build. When a user upgrades to the binding version that pins it:
- Runs read the new release by default.
- Stored model scores are still reused. A source's identity is not part of any score's lineage.
- Annotations stored under the source's identity, such as a writable annotation table, are cache misses for this source only, and are annotated again from the new release. Other sources are unaffected.
- An owned Parquet copy (
dataset) keeps serving the release its files record. - An owned BigQuery table (
table) is checked against the new release by default, so a table loaded from the old release is refused by its release labels; see Read from BigQuery. Set the source'sreleaseoption to the old release to keep reading it, or load the new release withaltar pull --bigquery.
An old release stays readable by its URI, as an owned Parquet dataset read anonymously:
[sources.clinvar]
type = "clinvar"
options = { dataset = "gs://kundajelab-altar/clinvar/clinvar-20260928-hg38", storage_options = { token = "anon" } }
Replace a release's layout¶
A layout-only rebuild replaces a release's files in place, under the same prefix:
- Build the same release, with the same inputs, to a staging prefix such as
gs://kundajelab-altar/_build/clinvar-20260928-hg38/. The file names must match the published ones; the builders name each file after its input. -
Verify the staging copy as in step 4.
altar pullreads only a binding's published location, so check the staging prefix from Python with the location replaced:from dataclasses import replace from altar.sources import PublishedLocation, pull_dataset from altar_clinvar import PUBLISHED_DATASET staging = replace( PUBLISHED_DATASET, locations=(PublishedLocation("gcs", "gs://kundajelab-altar/_build/clinvar-20260928-hg38"),), ) pull_dataset(staging, "/scratch/clinvar-20260928-hg38-staging")The row count must equal the published release's. 3. Copy the staging files over the published ones, object by object:
gcloud storage rsync --recursive --checksums-only \ gs://kundajelab-altar/_build/clinvar-20260928-hg38 \ gs://kundajelab-altar/clinvar/clinvar-20260928-hg38Do not pass
--delete-unmatched-destination-objects. The file names are the same, so nothing is unmatched. 4. Upload the copy from step 2 to the Hugging Face mirror withhf upload, in one commit. 5. Delete the staging prefix, and remove the build's write access.
Each object is replaced whole, but the copy takes time. While it runs, readers can see a mix of old and new files, all holding the same content. A source opened before the copy caches each file's size, footer, and the byte ranges it has read, so it can fail when it reads a replaced file. Reopen it, or restart a long-running process.
Per-dataset notes¶
ClinVar¶
- NCBI writes each release as dated files in
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/:
clinvar_<YYYYMMDD>.vcf.gzandclinvar_<YYYYMMDD>_papu.vcf.gz. Weekly files are inweekly/, and older releases move toarchive_2.0/<year>/. Build the dated files, never the undatedclinvar.vcf.gz, which NCBI replaces with each release. - Bump to each monthly release. Weekly releases are not published.
- Pass the
_papufile as a second input. It holds the variants ClinVar places on the chrY pseudoautosomal regions, and adds a second file to thechr=chrYpartition. --releaseis the date in the file names. The builder checks it against each file name and against the header's##fileDate, and refuses a mismatch.- Count the upstream records with
curl -s <url> | gzip -dc | grep -vc '^#'for each of the two files. - A warning about an unknown review status or
ORIGINbit means ClinVar added one. Decide whether the binding needs a change before publishing. - 20260928 on hg38: 4,555,530 alleles from 4,557,594 records, 26 files, 106 MB. The build took 2 min 45 s on a 64-core server, reading from NCBI and writing to the bucket. Only hg38 is published.
gnomAD¶
- gnomAD's releases are in its public bucket,
gs://gcp-public-data--gnomad/release/<version>/. The genomes sites VCFs arevcf/genomes/gnomad.genomes.v<version>.sites.chr<N>.vcf.bgz, 24 files for chr1 to chr22, X and Y. Pass all 24 as--input, with--dataset genomes --release <version>. - Build on a short-lived Compute Engine VM. 4.1.1 genomes was built on a 32-vCPU VM
with
--workers 24, one process per chromosome, in about 3 hours. The wall time is set by chr1, the largest input. - Each worker builds its input in local scratch under
TMPDIRbefore uploading it. Give the VM room for every file in flight: chr1's file is 1.9 GB, and with 24 workers the whole release, 24.9 GiB, can be in scratch at once. - Create the VM with a service account and the
cloud-platformscope. The default scopes give read-only storage access, and the upload fails. - Upstream record counts are only available by reading the 527 GiB of VCF, which the build does. Use the builder's summary, and compare rows per chromosome with the previous release.
- 4.1.1 genomes: 759,336,320 alleles, 24 files, 24.9 GiB.
- Delete the VM when the release is verified.
Open Targets E2G¶
- Open Targets publishes each Platform release, about once a quarter, as Parquet files at
https://ftp.ebi.ac.uk/pub/databases/opentargets/platform/<release>/output/enhancer_to_gene/. Releases are namedYY.MM, such as26.09. The builder reads 26.03 and later. -
Download the release's files first, then build from their directory. The EBI FTP site slows above a few parallel connections, so reading the files in place is slower:
-
The builder writes a local directory only. Upload it to the new prefix:
-
Count the upstream rows of every file, such as with DuckDB's
read_parquet. Links written plus identical repeats dropped must equal it. The builder stops on two rows that share anintervalIdand differ. The binding then needs a change. - 26.09: 48,808,491 rows from 30 files, 48,806,245 links written (2,246 identical repeats), 23 files, 2.70 GB. The build took 15 min 31 s on a 64-core server.
phyloP 100-way¶
- UCSC publishes one phyloP 100-way track for hg38 (
hg38.phyloP100way.bw, May 2015) and does not release new versions of it. Another track, such as phastCons or a wider alignment, or the same track on another genome build, is a new track rather than a release: see Add a conservation track. -
To rebuild the track, as for a layout change, download the bigWig named by the track's
upstream_urland build from it. The builder checks its MD5 againstupstream_md5: -
The builder writes a local directory only. Upload it with
gcloud storage rsync --recursive, and mirror it withhf upload-large-folder, since it is 5 GB. - Compare the rows per contig with the card's table. The bigWig holds 2,944,991,967 scored bases over 356 contigs; the build keeps chr1–22, X, Y and M.
- 2015-05 on hg38: 2,837,915,604 rows, 25 files, 5.04 GB. The build took 72 s and about 14 GB of memory on a 64-core server.
Access¶
- The bucket uses uniform bucket-level access, and anyone can read it. Everything in it is public.
- The account a build writes as needs object write access only during the build. Grant it
roles/storage.objectAdminon the bucket before the build, and remove the grant when the build is verified. A maintainer who builds with their own credentials needs the same access. - The bucket holds the builders' output only. Never put upstream downloads, source tarballs, credentials, or keys in it.
- Uploading to the Hugging Face mirror needs a token with write access to the
altar-genomicsorganization (hf auth login). Keep it on the machine that uploads.