Skip to content

Choose where results are stored

A run keeps each model's scores in a store. A later run that asks for the same scores reads them back from the store and starts no model. Four sections of a run spec decide where a run's results go:

Section What it keeps
[store] The score store. It holds each model's scores, one row per variant, which become the model's columns in the export.
[detail_store] A model's detail tables, with many rows per variant, such as one row per track. Without this section, they go beside the scores.
[artifact_store] Files a model writes and the run stores without reading, such as Sei's chromatin_profile_effects matrix.
[storage] How the run reads and writes files: inputs staged for container tasks, task outputs, the export and artifact files.

Annotation sources are not stored. The run looks them up again every time and records the release each one served. Where results live explains what each store keeps, and Score reuse and provenance when a stored score counts as the same computation and is reused.

Before you start

  • A spec that scores with a model; see Score variants with a model. The examples continue from the quickstart's score.toml.
  • For BigQuery: pip install "altar[bigquery]", a Google Cloud project, a BigQuery dataset that already exists, and Application Default Credentials, such as from gcloud auth application-default login.
  • For a Cloud Storage bucket: pip install "altar[gcs]". Other buckets need their own extra, such as altar[s3].

Keep scores in a SQLite file

A spec with a model and no [store] keeps its scores in a SQLite file in the cache, .altar/scores.sqlite. The cache is .altar in the directory you start altar run in, unless [run] cache_dir or the ALTAR_CACHE_DIR environment variable names another. The quickstart's score.toml uses this default.

  • The first run creates the cache and the file.
  • Every later run that uses the same cache reuses the scores in it, whatever its job_id or cohort. Start every run from the same directory, or set the same cache_dir.
  • Deleting a run's own directory, <job_id>/ by default, keeps the scores: they are in the cache.
  • A new file keeps lineages: a changed model setting adds a new set of scores beside the old ones under the same model key. Change a setting and rerun shows it.
  • A model's detail tables go to a second file beside the first, .altar/scores.details.sqlite.

A [store] section names another file:

[store]
type = "sqlite"
options = { path = "/data/altar/scores.sqlite" }
  • The first run creates the file, in a directory that must exist.
  • A relative path resolves against the directory you start altar run in. Start every run from the same directory, or write an absolute path.

Without path, the store is in memory. The scores are gone when the run ends, and the next run scores every variant again.

An annotation spec, with sources and no model, may leave out [store]. The run then assembles the export in memory and writes no store file, and creates no cache.

Keep scores in BigQuery

Preview

The BigQuery score and detail stores are tested on the BigQuery emulator and DuckDB. They have not been run against live BigQuery.

A bigquery store keeps the scores in a BigQuery table. It reaches BigQuery through a connection, a [connections.<name>] section that says which project to use and bill:

[connections.lab]
type = "gcp"
options = { project = "my-project" }

[store]
type = "bigquery"
connection = "lab"
options = { table_id = "my-project.altar.scores", lineage_table_id = "my-project.altar.score_lineages" }
  • table_id is the score table. The dataset must exist; the store creates the table on the first run.
  • lineage_table_id names a table that records each lineage. Without it, the store keeps no lineages, and a changed model setting under the same model key stops the run.
  • Queries run in the connection's project, and are billed to it.
  • Detail tables go to my-project.altar.scores__details, through the same connection.
  • The export is still written to <work_dir>/export on this machine.

Run at warehouse scale on BigQuery covers the tables the store creates, and joining annotation tables to the scores in SQL.

Put detail tables somewhere else

A model with detail tables, such as Enformer's track_effects, writes them on every run. [detail_store] names where they go when the default place beside the scores does not suit:

[detail_store]
type = "sqlite"
options = { path = "/data/altar/details.sqlite" }

The run opens the detail store only for a model that has detail tables. A bigquery detail store takes table_id and needs a connection, as the score store does. A variant counts as already scored only when the score store holds its scores and the detail store holds its detail rows, so keep both sections the same across runs.

Store artifacts

A model writes an artifact only when its output selects one; Select details and artifacts shows how. The spec then needs an [artifact_store]:

[artifact_store]
type = "storage"

The storage artifact store copies each file to <cache_dir>/artifacts and records it in a SQLite index, <cache_dir>/artifacts/index.sqlite, so a later run with the same cache finds them. root_uri puts the files elsewhere, and index_path the index.

Read and write files in a bucket

[storage] sets how the run reads and writes files. The default, local, reads and writes paths on this machine. fsspec reads and writes local paths too, and adds bucket URIs such as gs:// and s3://. With a connection, it uses that connection's project and credentials.

This spec keeps the artifact files in a Cloud Storage bucket, billed to my-project:

[connections.lab]
type = "gcp"
options = { project = "my-project" }

[storage]
type = "fsspec"
connection = "lab"

[artifact_store]
type = "storage"
options = { root_uri = "gs://my-bucket/altar/artifacts" }

The artifact index stays at <cache_dir>/artifacts/index.sqlite, on this machine. A Kubernetes backend copies its tasks' files through [storage] too; see Run on Kubernetes.

Check the result

altar run --validate names the score store:

  keep scores in sqlite (.altar/scores.sqlite), and write parquet to first-scores/export

It checks that the store's type is installed, but does not open the store: a BigQuery table it cannot reach stops the run, not --validate.

Run the same spec a second time. Each model's line counts the variants the store already held:

[models.legnet-wtc11] 5 candidates, 1 set aside (1 deletion; listed in first-scores/staging/jobs/first-scores/variants/legnet-wtc11/ineligible.tsv), 4 already scored, 0 scored in 0 batches, in 0.0s

The run summary's models entry gives the same count as cache_hit_count.

If it fails

  • A bigquery store without connection stops the run:

    altar run: error: store: 'bigquery' cannot be built from the spec's options (BigQueryScoreStore.__init__() missing 1 required positional argument: 'client'); it reaches its backend through a connection, so declare one and name it with store.connection
    

    Declare a gcp connection and name it with connection, as in Keep scores in BigQuery.

  • OperationalError: unable to open database file means the directory of a SQLite path does not exist. Create it, or leave out [store] to keep the scores in the cache.

  • A connection on a sqlite store, or on local storage, is refused, such as store.connection: 'sqlite' does not read through a connection; remove store.connection.
  • A connection that names no [connections.<name>] section is refused, with the names the spec declares.
  • A rerun that scores everything again found no stored scores it could use. The run scored everything again lists what to check.
  • A changed setting in a store without lineages stops the run; see A changed setting meets a store without lineages.

Next