Skip to content

Add a conservation track

A conservation track is one base-wise score file, such as a UCSC bigWig, on one genome build. altar-conservation turns each track into an annotation source that adds one column to every variant: the score at the variant's base. phyloP 100-way on hg38 is the first track. This page adds another, using UCSC's hg38 phastCons 100-way as the example.

A track needs:

  1. a contract and a ConservationTrack entry in tracks.py;
  2. the entry in TRACKS;
  3. a four-line source subclass;
  4. an entry point;
  5. tests;
  6. a build with altar-conservation-build --track;
  7. a published copy, following the release runbook;
  8. a Hugging Face dataset card.

The position score layout that a track is read from serves hg19 and hg38, and the builder keeps chr1–22, X, Y and M. A track on another assembly or with other contigs needs changes in altar first.

1. Declare the track

In bindings/conservation/altar_conservation/tracks.py, declare the track's contract, its one float column, and its ConservationTrack:

PHASTCONS100WAY_ANNOTATIONS = AnnotationContract(
    source_id="org.kundajelab.altar.annotation.phastcons100way",
    schema_version="1.0.0",
    columns=(AnnotationColumn(name="phastcons100way", dtype="float", label="phastCons 100-way"),),
)
"""The phastCons 100-way annotation contract: one float column, ``phastcons100way``.

The value is UCSC's phastCons score over the 100-vertebrate alignment at the variant's base: the posterior
probability, from 0 to 1, that the base is in a conserved element.
"""

PHASTCONS100WAY = ConservationTrack(
    contract=PHASTCONS100WAY_ANNOTATIONS,
    genome_build="hg38",
    release="2015-05",
    upstream_url="https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons100way/hg38.phastCons100way.bw",
    upstream_md5="8df8387aba88f4ca9eacdebb2e729745",
    published=_published(
        PHASTCONS100WAY_ANNOTATIONS, release="2015-05", genome_build="hg38", path="phastcons100way-hg38"
    ),
)
"""UCSC hg38 phastCons 100-way (``hg38.phastCons100way.bw``, May 2015), published as ``phastcons100way-hg38``."""
  • The column's name is the track's name. It is also the source's registry name and the builder's --track value.
  • release is the upstream file's date, YYYY-MM, from UCSC's directory listing.
  • upstream_md5 is the checksum the provider publishes, here from UCSC's md5sum.txt. The builder refuses a file that does not match.
  • path names the published copy, gs://kundajelab-altar/conservation/<path> and hf://datasets/altar-genomics/<path>. Follow the naming rules: the track's name and its genome build.

2. Add it to TRACKS

TRACKS lists every track by name. The builder's --track choices come from it:

TRACKS: dict[str, ConservationTrack] = {track.name: track for track in (PHYLOP100WAY, PHASTCONS100WAY)}

Add the two new names to the module's __all__.

3. Subclass the source

In source.py, a track's source is a subclass that names the track and the registry name:

class PhastCons100wayAnnotationSource(ConservationAnnotationSource):
    """UCSC hg38 phastCons 100-way (`PHASTCONS100WAY`), registered as ``phastcons100way``."""

    name = "phastcons100way"
    track = PHASTCONS100WAY

The subclass inherits from_options, the published copy, altar sources' contract and the option errors. ConservationAnnotationSource.__init_subclass__ sets supported_genome_builds to the track's genome build, so altar run refuses the source for a cohort on another build, and results assembled from Python join it only for variants on that build.

Export the subclass, the track and its contract from altar_conservation/__init__.py.

4. Register an entry point

In bindings/conservation/pyproject.toml, the entry point's key must equal the source's name:

[project.entry-points."altar.annotation_sources"]
phylop100way = "altar_conservation:PhyloP100wayAnnotationSource"
phastcons100way = "altar_conservation:PhastCons100wayAnnotationSource"

Reinstall the binding so the entry point is registered, for example with uvx uv@0.12.17 sync --all-packages --extra test --extra kubernetes.

5. Test it

In bindings/conservation/tests/test_conservation.py, mirror the phyloP tests for the new track:

  • the track and its published dataset: TRACKS, the registry, the contract, both location URIs, the identity and cluster_by;
  • supported_genome_builds is the track's build;
  • a conformance class over AnnotationSourceContract, and one over PublishedDatasetContract with reads_bigquery_tables returning False, both reading a dataset written with write_contig and the track's track_metadata;
  • altar sources lists the track, and altar sources <name> shows its release, build, URI and column.

Then run the binding's tests and the gate:

.venv/bin/python -m pytest bindings/conservation/tests -q
.venv/bin/python scripts/check_local.py

6. Build the dataset

Download the bigWig and build from it on a server with enough memory. phyloP 100-way took 72 s and about 14 GB on a 64-core server:

curl -O https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons100way/hg38.phastCons100way.bw
altar-conservation-build --track phastcons100way --input hg38.phastCons100way.bw \
  --output /scratch/phastcons100way-hg38

The builder checks the MD5, converts each primary contig in its own process, and records the track, release and genome build in every footer.

7. Publish it

Upload the build to its prefix and verify it, as Publish a dataset release describes, with the phyloP notes for a local build: gcloud storage rsync --recursive to gs://kundajelab-altar/conservation/phastcons100way-hg38, then altar pull phastcons100way --to <scratch> to check every file. Add the track to the runbook's table of published datasets.

8. Write the dataset card

Dataset cards live in bindings/conservation/hf-card/; README.md there is the phyloP card. Copy it to a card named for the new published copy, such as hf-card/phastcons100way-hg38.md, and update every track-specific value: the front matter, the title, what the score means, the upstream file and checksum, the row count, file count and size, the rows per contig, the footer values, the URIs, the DuckDB example, and the citation UCSC asks for. Copy the card into the build directory as README.md, then mirror the build to Hugging Face, as step 5 of the runbook shows.

Finally, document the track for users:

  • its own page under altar/docs/integrations/, on the integration page template, like phyloP 100-way, with an entry in the nav of altar/mkdocs.yml. Run scripts/gen_integration_pages.py --update to fill its generated blocks and add its row to the catalog;
  • the binding's README.