Package {datom}

datom: A Unified Framework for Versioned, Traceable Tabular Data

Description

Provides versioned storage for tabular data without a database or a server. Each table is written as an immutable, content-addressed version – identical content is detected and stored only once – while its version history and metadata are kept as code in a 'git' repository and the data itself in a local filesystem or cloud object storage ('S3'). Any past version can be read back exactly by its identifier, and each table records the sources it was derived from, so a project carries full data lineage. A lightweight reader role retrieves current or historical data from storage alone, without 'git' or write access, giving downstream analyses and pipelines a single versioned source of truth. It targets analytical and scientific data management, such as preparing clinical study datasets, and is designed as a foundation for higher-level governance tooling.

Author(s)

Maintainer: Afshin Mashadi-Hossein amashadihossein@gmail.com [copyright holder]

See Also

Useful links:

• https://github.com/amashadihossein/datom

• https://amashadihossein.github.io/datom/

• Report bugs at https://github.com/amashadihossein/datom/issues

Abbreviate SHA Hash

Description

Truncates a SHA-256 hash to a short prefix for display. Accepts character vectors; NA values pass through unchanged.

Usage

.datom_abbreviate_sha(sha, n = 8L)

Arguments

Value

Character vector of abbreviated hashes.

Build a datom_conn from Store Components

Description

Backend-aware helper that creates the appropriate client (S3 client or NULL) and assembles a datom_conn. Used by datom_init_repo(), .datom_get_conn_developer(), and .datom_get_conn_reader().

Usage

.datom_build_init_conn(
  project_name,
  data_store,
  path,
  role,
  endpoint = NULL,
  gov_store = NULL,
  gov_local_path = NULL,
  data_repo_url = NULL,
  github_pat = NULL,
  github_api_url = NULL
)

Arguments

Value

A datom_conn object.

Build Metadata Object

Description

Constructs the metadata list for a table write, including auto-computed fields (data_sha, dimensions, colnames, timestamp, datom_version) and any user-supplied custom metadata.

Usage

.datom_build_metadata(
  data,
  data_sha,
  custom = NULL,
  table_type = "derived",
  size_bytes = NULL,
  parents = NULL,
  source_lineage = NULL,
  original_file_sha = NULL,
  column_hashes = NULL
)

Arguments

Value

Named list suitable for writing as metadata.json. Always carries hash_algo = "datom-cv1" and declares parquet_sha (left NULL here and populated by datom_write() after change detection, since the stored- object hash is not knowable until then; it is excluded from metadata_sha so this deferred assignment is safe).

Build Full S3 URI

Description

Convenience function that combines bucket and key into an S3 URI.

Usage

.datom_build_s3_uri(bucket, key)

Arguments

Value

Character string S3 URI.

Build S3 Object Key

Description

Constructs S3 keys from path components, inserting the ⁠datom/⁠ segment per the storage structure convention.

Usage

.datom_build_storage_key(prefix = NULL, ...)

Arguments

Details

Mapping from arguments to key, for reference:

("proj", "customers", "abc123.parquet")
  -> "proj/datom/customers/abc123.parquet"

("proj", "customers", ".metadata", "metadata.json")
  -> "proj/datom/customers/.metadata/metadata.json"

("proj", ".metadata", "dispatch.json")
  -> "proj/datom/.metadata/dispatch.json"

(NULL, "customers", "abc123.parquet")
  -> "datom/customers/abc123.parquet"

Value

Character string S3 key.

Compute the datom-cv1 Canonical Content Hash

Description

The I/O-free identity engine for datom-cv1. Computes data_sha from the in-memory logical values only – no parquet write, no CSV, no temp files, no as.data.frame() or coercion, and never invokes arrow. Columns are read via data[[i]] / names(data) and dimensions via nrow() / ncol(), so two frames with equal values hash identically regardless of container class (tibble vs data.frame vs grouped_df), row names, or arrow version.

Usage

.datom_canonical_hash(data)

Arguments

Details

Before encoding, every column is scanned through .datom_hash_recourse(); if any are unsupported the function aborts once, listing every offender with its class and canonical recourse. This fires during data_sha computation (step 1 of datom_write()), before any git or storage mutation, so a refusal leaves no partial state.

The final hash is sha256( "datom-cv1" || f64le(nrow) || f64le(ncol) || concat(col_digest_hex...) ).

Value

A list with data_sha (character) and column_hashes (an ordered list of list(name, sha) in column order, computed once and reused for both data_sha and the persisted column index).

Validate Data Store Reachability

Description

Checks that the data store at the ref-resolved location is reachable. For S3: HeadBucket. For local: dir_exists. Provides actionable error messages when data is unreachable after migration.

Usage

.datom_check_data_reachable(conn, migrated = FALSE)

Arguments

Value

Invisible TRUE on success. Warns on network error (offline use ok).

Check git2r Availability

Description

Aborts with a helpful message if git2r is not installed.

Usage

.datom_check_git2r()

Value

Invisible TRUE if available.

Check Local Branch is Current with Remote

Description

Fetches from the remote and compares local HEAD SHA against the upstream HEAD SHA. If the local branch is behind, aborts with a clear message telling the developer to pull first.

Usage

.datom_check_git_current(path, pat = NULL)

Arguments

Details

Does NOT auto-pull - lets the developer decide how to resolve.

Value

Invisible TRUE if the local branch is up to date.

Validate Git Remote Reachability

Description

Checks that the data git remote URL is reachable and that credentials work. Called at conn-construction time in .datom_get_conn_developer() alongside .datom_check_data_reachable().

Usage

.datom_check_git_reachable(conn)

Arguments

Details

Failure behaviour:

• No data_repo_url: returns invisibly (structural pass, no network needed).

• HTTPS, auth failure (HTTP 401/403): hard abort pointing to github_pat.

• HTTPS, URL not found (HTTP 404): hard abort.

• HTTPS, network error (timeout/DNS): warn-only (offline-tolerant).

• SSH, any error: warn-only (cannot reliably distinguish "no agent" from "offline").

• SSH, success: invisible TRUE.

Value

Invisible TRUE on success. Warns on network error (offline use ok).

Check Whether an S3 Namespace is Free

Description

Checks for the existence of .metadata/manifest.json in the target S3 namespace. If found, the namespace is occupied by an existing datom project. Returns TRUE if the namespace is free. Aborts with an actionable error if occupied, showing the existing project name when possible.

Usage

.datom_check_namespace_free(conn)

Arguments

Details

Uses head_object first (cheap) and only reads the manifest (via get_object) when the namespace is occupied, to extract the project name for the error message.

Value

Invisible TRUE if the namespace is free.

Check ref.json Matches Connection (Write-Time Guard)

Description

Re-resolves ref.json from the governance store and compares against the current connection's data location. Errors if they disagree, preventing writes to the wrong location after a migration.

Usage

.datom_check_ref_current(conn)

Arguments

Value

Invisible TRUE if current, or skips silently if no governance fields.

Human-Readable Class Label for a Column

Description

Renders the label shown for a column in the all-offenders abort bullets and in the datom_check_hashable() report: the collapsed class(x) string for an explicitly-classed column, or typeof(x) for an unclassed one (so a list column reads list, a complex column complex, and a units column units).

Usage

.datom_class_label(x)

Arguments

Value

A single character string.

Compute a Single Column's datom-cv1 Digest

Description

Encodes one column to its per-column SHA-256 hex digest for datom-cv1, as sha256( utf8(tag) || utf8(colname) || 0x00 || payload ). The tag is the kind returned by .datom_column_kind() and the payload is produced by the shared encoders. Labelled columns strip their class and attributes and re-dispatch on the bare underlying vector, so value labels never enter identity.

Usage

.datom_col_digest(name, x)

Arguments

Value

A 64-character SHA-256 hex string.

Classify a Column for Canonical Hashing

Description

The single supported-type classifier underneath the datom-cv1 hash. It returns the dispatch kind for a hashable column or NULL for an unsupported one. Both the all-offenders gate (.datom_hash_recourse()) and the per-column encoder in .datom_canonical_hash() consume this one function, so a column the gate accepts can never be one the encoder cannot encode.

Usage

.datom_column_kind(x)

Arguments

Details

Dispatch is evaluated in a fixed order (reordering can silently change hashes): bit64::integer64, factor, Date (incl. data.table::IDate), POSIXct, difftime/hms, data.table::ITime, then haven_labelled/labelled/labelled_spss (stripped to their underlying type and re-classified), then any other explicitly-classed column is refused, then unclassed atomics (logical/integer/double as "num", character as "chr"), and finally any other type is refused.

Detection uses inherits() / typeof() / is.object() class-string matching only – it adds no new package dependency (bit64, data.table, haven are recognised by their class strings, not by being loaded).

Value

One of the kind tags "i64", "chr", "date", "time", "drtn", "num" for a supported column, or NULL when unsupported.

Compute the datom-cv1 Content Hash of a Data Frame

Description

Thin wrapper over .datom_canonical_hash() returning only the scalar data_sha. Preserves the scalar-string contract for callers that need just the content hash (for example the datom_sync() self-lineage entry). Row and column order are significant; there is no sort option.

Usage

.datom_compute_data_sha(data)

Arguments

Value

Character SHA-256 data_sha.

Compute SHA-256 of Metadata

Description

Sorts fields by C-locale byte order (method = "radix") before hashing so the result is deterministic regardless of field insertion order and regardless of the host's LC_COLLATE (default collation sorts differ between C and e.g. en_US.UTF-8, which would otherwise make the same metadata hash differently on different machines).

Usage

.datom_compute_metadata_sha(metadata)

Arguments

Details

Volatile fields are excluded so that identical semantic content always produces the same SHA regardless of when or how it was serialized: created_at and datom_version (write-time provenance), parquet_sha and size_bytes (stored-object byte facts – both drift with the arrow version and must not re-enter identity), and column_hashes (a deterministic function of the same values that already fix data_sha). original_file_sha and hash_algo remain in the semantic set – a new source file or a new hash algorithm legitimately defines a new version.

Hashes a JSON canonical form rather than the R object directly. This ensures that metadata read back from JSON (e.g., from S3) produces the same SHA as metadata built in-memory, despite R type differences (integer vs double, character vector vs list) introduced by JSON round-tripping.

Value

Character SHA-256 hash.

Compute SHA-256 of an Input File's Raw Bytes

Description

Answers "have this input artifact's bytes changed?". This is the original_file_sha of the three-SHA identity model – distinct from data_sha (canonical logical content) and parquet_sha (stored bytes).

Usage

.datom_compute_original_file_sha(path)

Arguments

Value

Character SHA-256 hash.

Scope-Selecting Connection Accessor

Description

Returns the connection shaped for either the data or governance store. The storage dispatch layer (⁠.datom_storage_*⁠) reads conn$root, conn$prefix, and conn$client; this accessor swaps those fields when callers need to operate on the governance store.

Usage

.datom_conn_for(conn, scope = c("data", "gov"))

Arguments

Details

Single source of truth for "which store am I talking to right now?" – replaces ad-hoc conn$gov_client peeking and the prior .datom_gov_conn() helper.

Value

A datom_conn object scoped to the requested store.

Copy a Single Storage Object Between Two Connections

Description

Dispatches on the (from_backend, to_backend) pair. For local->local uses fs::file_copy; all other combos transfer raw bytes.

Usage

.datom_copy_one(from_conn, to_conn, rel_key)

Arguments

Value

Named list with key (character) and bytes (numeric).

Create a GitHub Repository

Description

Creates a new GitHub repository via the REST API. Handles both org and personal repos.

Usage

.datom_create_github_repo(
  repo_name,
  pat,
  org = NULL,
  private = TRUE,
  api_url = "https://api.github.com"
)

Arguments

Details

Safety guard:

• Repo doesn't exist → create, return URL

• Repo exists + empty → reuse, return URL

• Repo exists + has content → abort

Value

The clone URL of the created/reused repository.

Build governance.json Content

Description

Constructs the governance pointer list that is written to both the local git copy and the data-store mirror.

Usage

.datom_create_governance_json(gov_repo_url, gov_store, attached_at = NULL)

Arguments

Value

Named list suitable for serialisation to JSON.

Create Initial ref.json Content

Description

Builds the initial ref.json structure from the data store component. No previous entries on first creation.

Usage

.datom_create_ref(data_store)

Arguments

Value

A list suitable for JSON serialization.

Delete a GitHub Repository

Description

Deletes a GitHub repository via the REST API. Requires a PAT with the delete_repo scope.

Usage

.datom_delete_github_repo(repo_full, pat, api_url = "https://api.github.com")

Arguments

Value

Invisible TRUE on success; aborts on failure.

Encode a Character Payload for Canonical Hashing

Description

The character encoder for the chr column kind (character and factor columns). Emits a one-byte-per-row NA mask (0x01 where is.na(), 0x00 otherwise) followed by each value re-encoded to UTF-8 via enc2utf8() and NUL-terminated. The leading mask makes NA and the empty string "" distinguishable (both have an empty value section, but NA sets its mask byte). No Unicode normalization is applied, so NFC and NFD forms of the same text encode differently (a documented, benign limitation).

Usage

.datom_encode_character(x)

Arguments

Value

A raw vector: length(x) mask bytes followed by the NUL-terminated UTF-8 value bytes.

Encode a Numeric Payload for Canonical Hashing

Description

The single shared numeric encoder used by the num, date, time, and drtn column kinds of datom-cv1. Produces a fixed, platform-independent byte sequence: IEEE-754 doubles written little-endian regardless of host endianness, with three canonicalizations so that logically-equal values encode identically:

Usage

.datom_encode_numeric(x)

Arguments

Details

• Every NaN payload (e.g. 0/0, a signalling NaN, a negative NaN) is folded to the pinned canonical quiet NaN bit pattern 0x7ff8000000000000 (see .datom_nan_canonical).

• -0.0 is converted to +0.0.

• NA_real_ is preserved as its own distinct bit pattern – it is a specific NaN payload in R (high word 0x7ff00000, low word 1954), fixed by R itself and therefore portable, and is deliberately not folded into the canonical NaN, so NA_real_ and NaN encode differently.

No rounding is applied: doubles are encoded bit-exact.

Why the canonical NaN is written as bytes, not assigned as a value. Assigning R's NaN (d[nan_idx] <- NaN) folds NaN payloads but inherits the host's NaN sign bit: R's NaN is ⁠0x7ff8...⁠ on macOS/arm64 and ⁠0xfff8...⁠ on Linux/x86_64, because it comes from a C-level 0.0/0.0. That made data_sha platform-dependent for any table containing a NaN – caught by the CI golden matrix (the macOS job passed, the Linux job did not). Splicing the pinned bytes in directly removes the host from the equation, which is the whole premise of a canonical hash.

Value

A raw vector of 8 * length(x) bytes.

Build Connection from Local Repo + Store (Developer Path)

Description

Reads .datom/project.yaml for project identity and cross-checks against the store config. Uses the store for credentials.

Usage

.datom_get_conn_developer(path, store, endpoint = NULL)

Arguments

Value

A datom_conn object.

Build Connection from Store (Reader Path)

Description

Constructs a connection from a store object and project_name. Uses the data component of the store for S3 configuration.

Usage

.datom_get_conn_reader(store, project_name, endpoint = NULL)

Arguments

Value

A datom_conn object.

Get Author Info from Git Config

Description

Reads user.name and user.email from the repository's git config.

Usage

.datom_git_author(path)

Arguments

Value

Named list with name and email.

Get Current Branch

Description

Returns the name of the currently checked-out branch. Aborts on detached HEAD (datom requires a branch).

Usage

.datom_git_branch(path)

Arguments

Value

Branch name as a string.

Commit Changes

Description

Stages the specified files and creates a commit.

Usage

.datom_git_commit(path, files, message, staged_deletions = FALSE)

Arguments

Value

Commit SHA as a string.

Build Git Credentials for HTTPS Remotes

Description

Returns a git2r::cred_user_pass object when the remote URL is HTTPS and a PAT has been supplied. Returns NULL for SSH remotes or when pat is absent.

Usage

.datom_git_credentials(remote_url, pat = NULL)

Arguments

Details

The PAT must be supplied explicitly – datom does not read environment variables internally. Callers obtain the PAT from conn$github_pat, which is populated at conn-construction time from store$github_pat.

Value

A git2r::cred_user_pass object or NULL.

Ensure a Repo Has a Local Git Identity

Description

Sets user.name and user.email on the local config of repo so that git2r::default_signature(repo) succeeds even when the host has no global git identity (e.g. CI runners). Values are taken from global config when present; otherwise fallback constants are used.

Usage

.datom_git_ensure_local_identity(
  repo,
  fallback_name = "datom",
  fallback_email = "datom@noreply"
)

Arguments

Details

Idempotent: re-setting the same values is a no-op from git's perspective.

Value

Invisible repo.

Pull from Remote (Fetch + Merge)

Description

Fetches from the remote and merges upstream changes into the current branch. Aborts on merge conflicts - user must resolve manually. This is the primary defense against diverged histories.

Usage

.datom_git_pull(path, pat = NULL)

Arguments

Value

Invisible TRUE on success.

Push to Remote

Description

Pulls (fetch + merge) first to detect conflicts, then pushes. Aborts on merge conflicts – user must resolve manually per spec.

Usage

.datom_git_push(path, pat = NULL, pull_first = TRUE)

Arguments

Value

Invisible TRUE on success.

Get GitHub Username from PAT

Description

Calls GET /user to get the authenticated user's login.

Usage

.datom_github_username(pat, api_url = "https://api.github.com")

Arguments

Value

Username string.

Check Whether a Gov Clone Exists

Description

Returns TRUE if gov_local_path is a directory that looks like a git repository (contains a .git folder). Does not validate the remote URL.

Usage

.datom_gov_clone_exists(gov_local_path)

Arguments

Value

Logical scalar.

Initialise Gov Clone (Clone If Missing, Reuse If Present)

Description

Ensures a valid gov clone exists at gov_local_path:

Usage

.datom_gov_clone_init(gov_repo_url, gov_local_path, pat = NULL)

Arguments

Details

• If the path does not exist: clones gov_repo_url into gov_local_path.

• If the path exists and is a git repo with matching remote URL: reuses it silently (idempotent).

• If the path exists with a different remote URL: hard abort (collision).

• If the path exists but is not a git repo: hard abort.

Value

Invisible gov_local_path (character).

Open an Existing Gov Clone

Description

Returns a git2r repository handle for the gov clone at gov_local_path. Aborts if the path is not a valid git repository.

Usage

.datom_gov_clone_open(gov_local_path)

Arguments

Value

A git2r::repository object.

List Registered Project Names

Description

Returns the set of project names registered in the governance repo. When a local gov clone is available, lists directories under ⁠{gov_local_path}/projects/⁠ (offline-friendly, reflects last gov-clone refresh). Otherwise lists keys under ⁠projects/⁠ via the gov storage client and extracts unique top-level segments.

Usage

.datom_gov_list_projects(gov_conn, gov_local_path = NULL)

Arguments

Details

Skips entries that don't contain a ref.json (corrupt registry rows).

Value

Character vector of project names (sorted, may be empty).

Build Project-Scoped Path Within Gov Clone

Description

Returns ⁠{gov_local_path}/projects/{project_name}/⁠. This is where dispatch.json, ref.json, and migration_history.json live for a given project in the shared governance repo.

Usage

.datom_gov_project_path(gov_local_path, project_name)

Arguments

Value

An fs_path character scalar.

Validate Gov Clone Remote URL

Description

Reads the first configured remote from the gov clone and compares it against expected_url. Aborts if they differ. This prevents silently reusing a clone that points at a different governance repo.

Usage

.datom_gov_validate_remote(gov_local_path, expected_url)

Arguments

Details

URL comparison is normalised: trailing .git is stripped from both sides before comparison so ⁠https://github.com/org/acme-gov⁠ and ⁠https://github.com/org/acme-gov.git⁠ are treated as equivalent.

Value

Invisible TRUE.

Detect Changes Against Current Metadata

Description

Compares the proposed metadata_sha against the current version in S3. Returns the type of change detected.

Usage

.datom_has_changes(conn, name, new_data_sha, new_metadata_sha)

Arguments

Value

Named list with two elements: change_type – "none" (no change), "metadata_only" (data same, metadata changed), or "full" (data changed) – and current, the already-read current metadata (or NULL for a brand-new table). Returning current lets datom_write() reuse it (the metadata_only parquet_sha carry-forward and the revert-to-older history scan) without a second storage read.

Canonical Recourse String for an Unhashable Column

Description

The single source of truth for the remediation advice attached to an unsupported column. Returns NULL when .datom_column_kind(x) classifies the column as hashable, otherwise the canonical recourse string for the first matching offender category. Both datom_check_hashable() and the .datom_canonical_hash() all-offenders abort call this one function, so the checker's advice and the abort's advice can never diverge.

Usage

.datom_hash_recourse(x)

Arguments

Details

The column name and class are added by the caller (a checker row or an abort bullet); the strings here are type-scoped only. Detection order matters: POSIXlt (a list under the hood) is matched before the generic list rows; the nested-data-frame list row before the generic list row; and the class-specific rows (units, sfc, yearmon/yearqtr/chron) before the "other classed" fallback.

Value

NULL when x is hashable, otherwise a canonical recourse string.

Union and deduplicate source_lineage lists (internal wrapper)

Description

Thin wrapper retained for existing internal callers. Delegates to the exported datom_lineage_union().

Usage

.datom_lineage_union(lineage_lists)

Arguments

Value

Deduplicated list of source_lineage entries.

Delete a File from Local Storage

Description

Delete a File from Local Storage

Usage

.datom_local_delete(conn, key)

Arguments

Value

Invisible TRUE on success.

Delete All Files Under a Local Storage Prefix

Description

Removes the directory at root/{prefix}/datom/{prefix_key} and everything inside it. A missing prefix is a no-op.

Usage

.datom_local_delete_prefix(conn, prefix_key = NULL)

Arguments

Value

Invisibly, 1L if the directory was removed, 0L if not found.

Download File from Local Storage

Description

Copies a file from the store directory to a local path. Creates parent directories if needed.

Usage

.datom_local_download(conn, key, local_path)

Arguments

Value

Invisible TRUE on success.

Check if Local Storage Object Exists

Description

Check if Local Storage Object Exists

Usage

.datom_local_exists(conn, key)

Arguments

Value

TRUE or FALSE.

List Objects in Local Storage

Description

Lists files under a given prefix in the store.

Usage

.datom_local_list_objects(conn, prefix)

Arguments

Value

Character vector of relative keys (relative to conn$root).

Resolve a Storage Key to a Local Path

Description

Builds the full filesystem path from conn$root, conn$prefix, and the relative key segments.

Usage

.datom_local_path(conn, key)

Arguments

Value

An absolute filesystem path.

Read and Parse JSON from Local Storage

Description

Reads a JSON file from the store and parses it. Uses simplifyVector = FALSE to match S3 behavior.

Usage

.datom_local_read_json(conn, key)

Arguments

Value

Parsed R list.

Upload File to Local Storage

Description

Copies a local file to the store directory. Creates parent directories if needed.

Usage

.datom_local_upload(conn, local_path, key)

Arguments

Value

Invisible TRUE on success.

Write an R List to Local Storage as JSON

Description

Serializes data to JSON and writes to the store directory. Creates parent directories if needed.

Usage

.datom_local_write_json(conn, key, data)

Arguments

Value

Invisible TRUE on success.

Most-recent version_history parquet_sha for a data_sha

Description

Scans the developer's local version_history.json (newest-first) for the most recent entry whose data_sha matches and that carries a non-empty parquet_sha. Returns NULL when none is found – including the transitional period before task 5.1 persists parquet_sha into history entries, and for pre-cv1 histories. Reads the local git clone (offline-friendly); a stale clone is tolerated because the subsequent git push serializes concurrent writers (a behind clone fails to push before it can upload).

Usage

.datom_lookup_history_parquet_sha(conn, name, data_sha)

Arguments

Value

Character parquet_sha, or NULL.

Mask a Secret for Display

Description

By default shows the first 4 characters followed by ⁠****⁠. That prefix is fine for GitHub PATs (the ghp_/github_pat_ prefix is a public type tag), but for AWS secret access keys and session tokens the first characters are real entropy – pass reveal_prefix = FALSE to mask them fully.

Usage

.datom_mask_secret(secret, reveal_prefix = TRUE)

Arguments

Value

Masked string.

Normalize a prefix value to NULL or a non-empty string

Description

A NULL prefix serializes to JSON as an empty object () and reads back as an empty list, not NULL. Empty strings can also creep in. This collapses all empty-ish forms (NULL, list(), "", NA) to NULL so that location equality checks survive a JSON round-trip.

Usage

.datom_normalize_prefix(prefix)

Arguments

Value

NULL or a single non-empty character string.

Parse a ref.json structure into a location list

Description

Common parsing logic shared by storage-backed and clone-backed ref readers.

Usage

.datom_parse_ref(ref, source)

Arguments

Value

A named list with root, prefix, region.

Parse S3 URI into Components

Description

Extracts bucket and prefix from an ⁠s3://⁠ URI.

Usage

.datom_parse_s3_uri(uri)

Arguments

Details

Mapping from URI to components, for reference:

"s3://my-bucket/data/proj" -> list(bucket = "my-bucket", prefix = "data/proj")
"s3://my-bucket"           -> list(bucket = "my-bucket", prefix = NULL)

Value

Named list with bucket (character) and prefix (character or NULL).

Push Metadata Files to S3

Description

Uploads metadata.json, version_history.json, and a versioned snapshot to S3. Called AFTER git commit+push succeeds to maintain local → git → S3 ordering.

Usage

.datom_push_metadata_s3(conn, name, metadata, metadata_sha)

Arguments

Value

Invisible character vector of S3 keys written.

Read governance.json from Local Git Clone

Description

Reads and validates {path}/.datom/governance.json. Returns NULL when the file is absent (project is not gov-attached). Aborts on malformed JSON or failed schema validation.

Usage

.datom_read_governance_json_local(path)

Arguments

Value

Parsed list or NULL.

Read Table Metadata from S3

Description

Fetches both metadata.json (current state) and version_history.json (version index) for a given table from S3.

Usage

.datom_read_metadata(conn, name)

Arguments

Value

Named list with current (metadata.json contents) and history (version_history.json contents as a list of entries).

Download and Read Parquet from S3

Description

Downloads ⁠{table}/{data_sha}.parquet⁠ from S3 to a temporary file and reads it via arrow::read_parquet(). When an expected parquet_sha is supplied (non-empty), the downloaded object's SHA-256 is verified against it BEFORE parsing, so corruption or tampering aborts rather than being silently read.

Usage

.datom_read_parquet(conn, name, data_sha, parquet_sha = NULL)

Arguments

Value

Data frame.

Render README.md from Template

Description

Reads the template from inst/templates/README.md and fills in project-specific values using {{{ }}} delimiters.

Usage

.datom_render_readme(
  project_name,
  backend = "s3",
  root,
  prefix,
  region = NULL,
  remote_url,
  gov = NULL
)

Arguments

Value

Character string — the rendered README content.

Require Governance Attached on a Connection

Description

Guard helper used by gov-only commands (e.g. datom_projects) to fail with a single uniform message when called on a no-governance connection.

Usage

.datom_require_gov(conn, what)

Arguments

Value

Invisible TRUE when gov is attached. Aborts otherwise.

Resolve Data Location via Ref (Conn-Time Helper)

Description

Called during datom_get_conn() for both readers and developers when a governance store is present. Reads ref.json from governance, detects migration (store$data location != ref location), and returns the ref-resolved location.

Usage

.datom_resolve_data_location(
  store,
  role,
  project_name = NULL,
  path = NULL,
  gov_local_path = NULL,
  endpoint = NULL
)

Arguments

Details

Read path is role-aware:

• Developer with gov_local_path set: read projects/{name}/ref.json from the local gov clone (faster, works offline, reflects last gov-clone refresh).

• Otherwise (reader, or developer without a clone yet): read via the gov storage client.

Value

A named list with root, prefix, region from the ref, or NULL if no governance store is present (skip ref resolution).

Resolve the Local Path for the Governance Clone

Description

Returns the explicit override if supplied. Otherwise, places the gov clone as a sibling of data_local_path named after the basename of gov_repo_url (stripping a trailing .git suffix). This ensures the gov clone directory name reflects the gov repo's own identity, not any specific data project.

Usage

.datom_resolve_gov_local_path(data_local_path, gov_repo_url, override = NULL)

Arguments

Value

Absolute path string for the gov clone.

Resolve Gov Clone Path with Store Defaults

Description

Convenience wrapper that derives a gov clone path from a datom_store: returns the store's explicit gov_local_path if set; otherwise derives a sibling-of-data default from gov_repo_url; otherwise returns NULL.

Usage

.datom_resolve_or_default_gov_path(store, data_local_path)

Arguments

Details

Centralises the three-arm pattern previously duplicated in datom_init_repo(), datom_clone(), and .datom_get_conn_developer().

Value

Character path string or NULL.

Resolve the parquet_sha to Record and Whether to Upload

Description

For a write that is not a no-op, decides which parquet_sha the new metadata should carry and whether the freshly-serialized parquet bytes need uploading. The caller performs the actual upload AFTER the git push (git push is the serialization point); this function only decides.

Usage

.datom_resolve_parquet_sha(
  conn,
  name,
  data_sha,
  new_parquet_sha,
  change_type,
  current
)

Arguments

Details

Cases:

• metadata_only – the data_sha is unchanged, so the parquet object already exists; carry forward the current metadata's parquet_sha (which may be NULL for a pre-cv1 table, leaving the integrity check skipped) and do not upload.

• full where a prior version already recorded a parquet_sha for this exact data_sha – the stored object exists and is pinned by that version; reuse its parquet_sha and do NOT re-upload (a fresh serialization can differ byte-for-byte and would break that version's integrity pin).

• full otherwise (brand-new content, or a legacy object with no recorded parquet_sha) – upload these bytes and record their hash.

This refines the design's literal step 7 (which gated on .datom_storage_exists()): a recorded parquet_sha is the precise thing we must not clobber, and its presence implies the object exists, so the history lookup subsumes the existence check with identical behavior and one fewer storage round-trip.

Value

List with parquet_sha (character or NULL) and upload (logical).

Resolve Data Location from Governance Store

Description

Reads projects/{project_name}/ref.json from the governance store and returns the current data location as a named list. Single read, no recursion, no chain-walking.

Usage

.datom_resolve_ref(gov_conn, project_name = NULL)

Arguments

Details

If the ref has previous entries, a deprecation-style warning is emitted to alert users that a migration occurred and old locations may sunset.

Value

A named list with root, prefix, region for the current data location.

Resolve Data Location from Local Gov Clone

Description

Reads projects/{project_name}/ref.json directly from a local gov clone on disk. Faster than storage reads, works offline, and reflects the last gov-clone refresh. Used for developer connections.

Usage

.datom_resolve_ref_from_clone(gov_local_path, project_name)

Arguments

Value

A named list with root, prefix, region for the current data location.

Resolve Version to data_sha and parquet_sha

Description

Given metadata from .datom_read_metadata(), resolves a version spec to the corresponding data_sha (the storage address) and the recorded parquet_sha (the stored-object integrity hash). If version is NULL, resolves from the current metadata.json; if a metadata_sha string, looks it up in version_history.json.

Usage

.datom_resolve_version(metadata_list, version = NULL, name = "table")

Arguments

Details

The parquet_sha may be NULL/"" for pre-cv1 metadata, and for any version-pinned read until version_history entries persist parquet_sha (task 5.1). A NULL/empty parquet_sha tells .datom_read_parquet() to skip the integrity check (the intended pre-cv1 grace).

Value

Named list with data_sha (character) and parquet_sha (character or NULL) for the resolved version.

Create an S3 Client from Credentials

Description

Constructs a paws.storage::s3() client from credential values. Never stores raw credentials beyond the paws client object.

Usage

.datom_s3_client(
  access_key,
  secret_key,
  region = "us-east-1",
  endpoint = NULL,
  session_token = NULL
)

Arguments

Value

A paws.storage S3 client.

Delete All S3 Objects Under a Prefix

Description

Lists every key under {prefix}/datom/{prefix_key} and deletes in batches of up to 1000. A missing prefix is a no-op.

Usage

.datom_s3_delete_prefix(conn, prefix_key = NULL)

Arguments

Value

Invisibly, the count of deleted objects.

Download File from S3

Description

Downloads an S3 object and writes it to a local path. Creates parent directories if needed.

Usage

.datom_s3_download(conn, s3_key, local_path)

Arguments

Value

Invisible TRUE on success.

Check if S3 Object Exists

Description

Uses a HEAD request for efficiency. Returns TRUE if the object exists, FALSE on 404/NoSuchKey. Any other error (403, network) is re-thrown.

Usage

.datom_s3_exists(conn, s3_key)

Arguments

Value

TRUE or FALSE.

List S3 Objects Under a Prefix

Description

Lists every key under {prefix}/datom/{prefix_key} and returns relative keys (relative to the datom namespace, i.e. with the ⁠prefix/datom/⁠ part stripped). Paginates via ContinuationToken.

Usage

.datom_s3_list_objects(conn, prefix)

Arguments

Value

Character vector of relative keys (may be empty).

Read and Parse JSON from S3

Description

Downloads an S3 object, reads it as text, and parses as JSON. Uses simplifyVector = FALSE to keep lists as lists (matching how .datom_s3_write_json() writes them).

Usage

.datom_s3_read_json(conn, s3_key)

Arguments

Value

Parsed R list.

Upload File to S3

Description

Reads a local file as raw bytes and uploads via put_object().

Usage

.datom_s3_upload(conn, local_path, s3_key)

Arguments

Value

Invisible TRUE on success.

Write an R List to S3 as JSON

Description

Serializes data to JSON via jsonlite::toJSON() and uploads to S3.

Usage

.datom_s3_write_json(conn, s3_key, data)

Arguments

Value

Invisible TRUE on success.

Get Byte Size of a Single Storage Object

Description

Returns the byte size of the object at rel_key without reading its content. For S3 uses HEAD; for local uses fs::file_size(). Errors if the object is not found.

Usage

.datom_storage_byte_size(conn, rel_key)

Arguments

Value

Numeric byte count.

Compute SHA-256 Hash of a Storage Object's Content

Description

For S3, downloads the raw bytes and hashes in memory. For local, hashes the file directly. Used by datom_storage_verify() in content mode.

Usage

.datom_storage_content_hash(conn, rel_key)

Arguments

Value

Character SHA-256 hex string.

Delete governance.json Mirror from Data Storage

Description

Removes the governance.json mirror during project teardown. No-ops silently when the key is absent. Deletion is implemented via prefix-delete on the exact key path.

Usage

.datom_storage_delete_governance_json(conn)

Arguments

Value

Invisible NULL.

Delete All Objects Under a Storage Prefix

Description

Removes every file under prefix/datom/{prefix_key} from storage. For S3 this lists then batch-deletes. For local it removes the directory. A missing prefix is a no-op (returns 0L). Pass prefix_key = NULL to delete the entire datom namespace for this connection.

Usage

.datom_storage_delete_prefix(conn, prefix_key = NULL)

Arguments

Value

Invisibly, the count of deleted objects.

Download File from Storage

Description

Download File from Storage

Usage

.datom_storage_download(conn, key, local_path)

Arguments

Value

Invisible TRUE on success.

Check if Storage Object Exists

Description

Check if Storage Object Exists

Usage

.datom_storage_exists(conn, key)

Arguments

Value

TRUE or FALSE.

List Objects Under a Storage Prefix

Description

Returns the keys of every object under {prefix}/datom/{prefix_arg}. Keys are returned in their full storage-key form (i.e. including the ⁠{prefix}/datom/⁠ portion), matching what .datom_local_list_objects() and .datom_s3_list_objects() return.

Usage

.datom_storage_list_objects(conn, prefix)

Arguments

Value

Character vector of full storage keys (may be empty).

Read governance.json Mirror from Data Storage

Description

Returns the parsed list, or NULL when the key is absent. Aborts on any non-not-found storage error or on failed schema validation.

Usage

.datom_storage_read_governance_json(conn)

Arguments

Value

Parsed list or NULL.

Read and Parse JSON from Storage

Description

Read and Parse JSON from Storage

Usage

.datom_storage_read_json(conn, key)

Arguments

Value

Parsed R list.

Strip datom Namespace Prefix from a Full Storage Key

Description

Converts a full storage key (as returned by .datom_storage_list_objects()) to a relative key suitable for upload/download helpers (after ⁠{prefix}/datom/⁠).

Usage

.datom_storage_rel_key(full_key, conn)

Arguments

Value

Relative key string.

Upload File to Storage

Description

Upload File to Storage

Usage

.datom_storage_upload(conn, local_path, key)

Arguments

Value

Invisible TRUE on success.

Write governance.json Mirror to Data Storage

Description

Writes content to .metadata/governance.json in the data store. Uses .datom_storage_write_json() dispatch (backend-neutral).

Usage

.datom_storage_write_governance_json(conn, content)

Arguments

Value

Invisible NULL.

Write an R List to Storage as JSON

Description

Write an R List to Storage as JSON

Usage

.datom_storage_write_json(conn, key, data)

Arguments

Value

Invisible TRUE on success.

Get Backend Type from Store Component

Description

Get Backend Type from Store Component

Usage

.datom_store_backend(component)

Arguments

Value

"s3" or "local".

Build a Store-Constructor Snippet for a Component

Description

Renders a copy/paste datom_store_local(...) or datom_store_s3(...) call string for a store component, for embedding in a generated README. Secrets are shown as placeholders.

Usage

.datom_store_constructor_snippet(component)

Arguments

Value

Character scalar — an R constructor call as text.

Get Region from Store Component

Description

Returns the AWS region for S3, NULL for local.

Usage

.datom_store_region(component)

Arguments

Value

Region string or NULL.

Get Root from Store Component

Description

Returns the storage root: bucket name for S3, directory path for local.

Usage

.datom_store_root(component)

Arguments

Value

Root string.

Sync Data-Side Metadata to Storage

Description

Mirrors the data repo's metadata to the data store so readers see current state: the manifest (.metadata/manifest.json) and each table's metadata ({name}/.metadata/metadata.json, version_history.json).

Usage

.datom_sync_data_metadata(conn, .confirm = TRUE)

Arguments

Details

Data-only: governance files (dispatch.json, ref.json, migration_history.json) are not touched here. Governance sync is owned by the governance layer (gov_sync_dispatch()).

Used after a failed upload, or by datom_validate(fix = TRUE), to bring storage metadata back in line with the local data clone. Requires a developer connection with a local repo path.

Value

Invisibly, a list with repo_files (character vector of synced keys) and tables (list of per-table sync results).

Sync governance.json Storage Mirror from Git Copy

Description

Reads the git-canonical copy and overwrites the storage mirror. Call after a partial failure to repair a missing or stale storage mirror.

Usage

.datom_sync_governance_json(conn)

Arguments

Value

Invisible NULL.

Sync Single Table Metadata to S3

Description

Sync Single Table Metadata to S3

Usage

.datom_sync_metadata(conn, name)

Arguments

Value

Summary of sync operation.

Validate GitHub PAT

Description

Calls GitHub GET /user to verify the PAT is valid.

Usage

.datom_validate_github_pat(pat, api_url = "https://api.github.com")

Arguments

Value

A list with login and id.

Validate a datom Table Name

Description

Checks that a table name is filesystem-safe and S3-safe. Returns the name invisibly on success, errors with a clear message on failure.

Usage

.datom_validate_name(name)

Arguments

Value

Invisible name on success.

Validate parents Field Structure

Description

Checks that parents is either NULL or a list of entries each containing non-empty string fields source, table, version, and data_sha. WHERE an entry carries a non-NULL, non-empty source_lineage field, it is validated via .datom_validate_source_lineage(). Aborts with a cli error pointing to the first invalid entry.

Usage

.datom_validate_parents(x)

Arguments

Value

Invisibly TRUE if valid.

Validate S3 Store Connectivity

Description

Checks bucket access via HeadBucket. This validates both credentials and bucket existence/permissions in a single call.

Usage

.datom_validate_s3_store(access_key, secret_key, session_token, region, bucket)

Arguments

Value

Invisible TRUE on success.

Validate a SHA-Like Input (Version / data_sha)

Description

Ensures a user-supplied SHA-like string is 6-64 lowercase hex characters. Used to guard values that get spliced into a storage key ({table}/{sha}) – on the local backend an unvalidated value like "../../x" would escape the namespace via fs::path(). The 6-char minimum still covers the short prefixes .datom_resolve_version() intentionally accepts.

Usage

.datom_validate_sha(x, arg = "version")

Arguments

Value

Invisible x on success. Aborts otherwise.

Validate source_lineage Field Structure

Description

Checks that source_lineage is either NULL or a list of entries each containing non-empty string fields project, table, and version_sha. Extra fields are allowed (pass-through). Aborts with a cli error pointing to the first invalid entry.

Usage

.datom_validate_source_lineage(x)

Arguments

Value

Invisibly TRUE if valid.

Verify a Single Storage Object

Description

Checks that the object at rel_key in to_conn matches the one in from_conn. Returns a named list with key, ok (logical), and issue (character or NA_character_).

Usage

.datom_verify_one(from_conn, to_conn, rel_key, mode)

Arguments

Value

Named list: key, ok, issue.

Write governance.json to Local Git Clone

Description

Writes content to {path}/.datom/governance.json. The directory must already exist (created during datom_init_repo() or datom_repo_attach_governance()).

Usage

.datom_write_governance_json_local(path, content)

Arguments

Value

Invisible NULL.

Write Metadata Files to Git and S3 (Legacy Wrapper)

Description

Calls .datom_write_metadata_local() then .datom_push_metadata_s3(). Kept for backward compatibility. Does NOT commit or push.

Usage

.datom_write_metadata(conn, name, metadata, metadata_sha, message = NULL)

Arguments

Value

Invisible list with metadata_sha, git_paths, and s3_keys.

Write Metadata Files Locally

Description

Writes metadata.json and appends to version_history.json in the local git repo. Does NOT commit, push, or touch S3 — the caller handles those.

Usage

.datom_write_metadata_local(
  conn,
  name,
  metadata,
  metadata_sha,
  message = NULL,
  original_file_sha = NULL
)

Arguments

Value

Invisible list with metadata_sha and local paths written.

Check if Object is a Store Component

Description

Returns TRUE for any datom store component type (datom_store_s3, future datom_store_local, etc.).

Usage

.is_datom_store_component(x)

Arguments

Value

TRUE or FALSE.

Check Whether a Table Can Be Hashed by datom

Description

Pre-flight check for the datom table contract. Reports, per column, whether datom_write() can hash it and – when it cannot – exactly what to do about it. Run this before a write to fix a table in one pass instead of discovering offenders one error at a time.

Usage

datom_check_hashable(data)

Arguments

Details

datom identifies a table version by a canonical hash of its contents (data_sha), which requires every column to be a supported type: logical, integer, double, character, factor, Date, POSIXct, difftime/hms, data.table::ITime/IDate, bit64::integer64, or a labelled vector over one of those. List columns (including nested data frames, blobs, and POSIXlt), complex, raw, sf geometry, units, and zoo/chron columns are refused with specific advice.

The advice printed here is the same single-source recourse text datom_write() would abort with, so the two can never disagree.

Value

Invisibly, a data frame with one row per column of data and columns:

column

Column name.

class

Collapsed class string, or typeof() when unclassed.

status

"ok" or "unsupported".

recourse

NA when ok, otherwise how to make the column hashable.

See Also

datom_write()

Examples

# A clean table: every column is a supported type
clean <- data.frame(
  id = 1:3,
  score = c(1.5, 2.5, 3.5),
  label = c("a", "b", "c"),
  grp = factor(c("x", "y", "x")),
  day = as.Date(c("2026-01-01", "2026-01-02", "2026-01-03"))
)
datom_check_hashable(clean)

# An offending table: a list column and a complex column
messy <- data.frame(id = 1:2)
messy$notes <- list(c("a", "b"), "c")
messy$z <- c(1 + 2i, 3 + 4i)
report <- datom_check_hashable(messy)
report[report$status == "unsupported", c("column", "recourse")]

Clone a datom Repository

Description

Clones a remote datom repository and returns a connection. This is the recommended way for teammates to join an existing datom project – it wraps git2r::clone() and immediately returns a ready-to-use datom_conn.

Usage

datom_clone(path, store, ...)

Arguments

Details

When store$gov_repo_url is set the governance repo is also cloned (or verified if it already exists locally). An existing clone with uncommitted changes causes an error to avoid surprising state.

Value

A datom_conn object (developer role).

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)

  # A teammate joins the project from the remote alone.
  conn <- datom_clone(path = file.path(tmp, "teammate"), store = store)
  print(datom_list(conn))

  unlink(tmp, recursive = TRUE)
}

Monthly Cutoff Dates for Example Study

Description

Returns a named vector of monthly cutoff dates for STUDY-001, useful for simulating EDC data evolution in examples.

Usage

datom_example_cutoffs()

Value

Named character vector with entries month_1 through month_6.

Examples

datom_example_cutoffs()
# month_1    month_2    month_3    month_4    month_5    month_6
# "2026-01-28" "2026-02-28" ...

Load Example EDC Data

Description

Loads bundled clinical trial example data for use in examples and vignettes. The data simulates a Phase II study (STUDY-001) with 48 subjects enrolled over 6 months across four SDTM-flavored domains.

Usage

datom_example_data(domain = c("dm", "ex", "lb", "ae"), cutoff_date = NULL)

Arguments

Value

A data frame.

Examples

# Full demographics
dm <- datom_example_data("dm")

# Month-3 snapshot (subjects enrolled by 2026-03-28)
dm_m3 <- datom_example_data("dm", cutoff_date = "2026-03-28")

# Labs collected through Month 3
lb_m3 <- datom_example_data("lb", cutoff_date = "2026-03-28")

Get a datom Connection

Description

Flexible connection for both developers and readers.

Usage

datom_get_conn(path = NULL, store = NULL, project_name = NULL, endpoint = NULL)

Arguments

Details

Developer (local repo + store): provide path and store. Reads project identity from .datom/project.yaml; uses store for credentials and S3 config. Cross-checks bucket/prefix between yaml and store.

Reader (no local repo): provide store and project_name. Store provides everything.

Value

A datom_conn object.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)

  # Developer: local repo plus store.
  conn <- datom_get_conn(path = file.path(tmp, "repo"), store = store)
  print(conn)

  # Reader: store plus project name, no local repo.
  reader_store <- datom_store(data = datom_store_local(file.path(tmp, "storage")))
  print(datom_get_conn(store = reader_store, project_name = "example_project"))

  unlink(tmp, recursive = TRUE)
}

Get Lineage for a Table

Description

Reads lineage metadata for a table. Depending on depth, returns either the pre-computed transitive source list (source_lineage) or the immediate parent list (parents). Both fields are stored flat in the table's metadata – no walking or cross-project resolution is performed.

Usage

datom_get_lineage(conn, name, version = NULL, depth = c("source", "parents"))

Arguments

Details

The two fields answer different questions:

• "source": "what raw datasets does this table ultimately depend on?" (audit, regulatory disclosure, reproducibility scope). Derived at write time as the deduplicated union of the parents' source_lineage fields.

• "parents": "what did this table come from one step back?" (debugging, diff, replay). Equivalent to datom_get_parents().

Value

For depth = "source": the table's recorded source_lineage – a list of source-table descriptors (each with project, table, version_sha), or NULL if the field is absent. For depth = "parents": list of parent entries (each with source, table, version, data_sha), or NULL if no lineage is recorded.

See Also

datom_get_parents() for a direct shorthand for the "parents" depth.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")
  print(datom_get_lineage(conn, "dm", depth = "parents"))
  print(datom_get_lineage(conn, "dm", depth = "source"))

  unlink(tmp, recursive = TRUE)
}

Get Parent Lineage for a Table

Description

Reads the parents field from a table's metadata. Returns the lineage entries recorded at write time by datom_write(). For imported tables or derived tables with no recorded lineage, returns NULL.

Usage

datom_get_parents(conn, name, version = NULL)

Arguments

Value

List of parent entries (each with source, table, version, data_sha), or NULL if no lineage is recorded. The data_sha field is the parent's authoritative data SHA recorded via datom_parent(), and together with source and version is sufficient to select the parent's project connection and its pinned version.

See Also

datom_get_lineage() for a unified interface that also exposes the transitive source_lineage field via depth = "source".

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  dm <- datom_example_data("dm")
  datom_write(conn, data = dm, name = "dm")
  datom_write(
    conn,
    data    = dm[dm$SEX == "F", ],
    name    = "dm_female",
    parents = list(datom_parent(conn, "dm", datom_history(conn, "dm")$version[1]))
  )
  print(datom_get_parents(conn, "dm_female"))

  unlink(tmp, recursive = TRUE)
}

Show Version History

Description

Shows version history for a table by reading version_history.json from S3. Returns the most recent n versions.

Usage

datom_history(conn, name, n = 10, short_hash = FALSE)

Arguments

Value

Data frame with columns: version, data_sha, timestamp, author, commit_message.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")
  print(datom_history(conn, "dm"))

  unlink(tmp, recursive = TRUE)
}

Initialize a datom Repository

Description

One-time setup for data developers. Creates folder structure, initializes git with remote, sets up configuration files, and pushes to S3.

Usage

datom_init_repo(
  path = ".",
  project_name,
  store,
  create_repo = FALSE,
  repo_name = project_name,
  max_file_size_gb = 1000,
  git_ignore = c(".Rprofile", ".Renviron", ".Rhistory", ".Rapp.history", ".Rproj.user/",
    ".DS_Store", "*.csv", "*.tsv", "*.rds", "*.txt", "*.parquet", "*.sas7bdat", ".RData",
    ".RDataTmp", "*.html", "*.png", "*.pdf", ".vscode/", "rsconnect/"),
  .force = FALSE
)

Arguments

Details

Initializes the data repository only. The project is left as a solo project: project.yaml is the location authority, no governance.json / dispatch.json / ref.json is written, and project.yaml omits the storage.governance and repos.governance blocks. A governance store component on store, if present, is ignored here. Governance is attached later via the governance layer (gov_attach()).

Value

Invisible TRUE on success.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )

  datom_init_repo(
    path = file.path(tmp, "repo"),
    project_name = "example_project",
    store = store
  )
  print(list.files(file.path(tmp, "repo"), all.files = TRUE, no.. = TRUE))

  unlink(tmp, recursive = TRUE)
}

Union and deduplicate source_lineage lists

Description

Takes a list of zero or more source_lineage lists and returns their deduplicated union. Each entry is a list with project, table, and version_sha. Deduplication uses the composite key paste(project, table, version_sha, sep = "\t"), so each distinct entry appears exactly once and retained entries are returned unchanged.

Usage

datom_lineage_union(lineages)

Arguments

Details

NULL members are tolerated (a parent may carry source_lineage = NULL) and treated as an empty contribution. Empty input, or a list containing only empty lineage lists, returns an empty list.

This helper is the building block of the composable lineage recompute recipe. To check that a derived table's recorded source_lineage matches its parents, read each parent through a connection scoped to that parent's project and union their lineages:

# conn_c is scoped to the derived table's project.
parents <- datom_get_parents(conn_c, "c")

# One connection per project, keyed by each parent's `source`. Never
# reach across project stores with a single connection.
conns <- list(project_a = conn_a, project_b = conn_b)

# Read each parent's lineage through its own project connection.
parent_sls <- lapply(parents, function(p) {
  datom_get_lineage(conns[[p$source]], p$table, version = p$version,
                    depth = "source")
})

recomputed <- datom_lineage_union(parent_sls)
recorded   <- datom_get_lineage(conn_c, "c", depth = "source")
identical(recomputed, recorded)

Value

A deduplicated list of source_lineage entries, or an empty list when there is nothing to union.

Examples

sl1 <- list(list(project = "p", table = "t", version_sha = "a"))
sl2 <- list(list(project = "p", table = "t", version_sha = "a"))
datom_lineage_union(list(sl1, sl2))

List Available Tables

Description

Lists tables from S3 manifest. Reads .metadata/manifest.json from S3 and returns a data frame with one row per table.

Usage

datom_list(conn, pattern = NULL, include_versions = FALSE, short_hash = TRUE)

Arguments

Value

Data frame with table info (name, current_version, last_updated, etc.).

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")
  print(datom_list(conn))

  unlink(tmp, recursive = TRUE)
}

Declare a parent for lineage

Description

Resolves a parent table against a single project connection and returns a pure-data lineage record. The parent's authoritative data_sha and its source_lineage are read from the parent's own versioned metadata snapshot at ⁠{table}/.metadata/{version}.json⁠; a caller cannot supply or override data_sha (there is no data_sha parameter). The returned record retains no live connection and is serializable as plain data.

Usage

datom_parent(conn, table, version)

Arguments

Details

Same-project and cross-project parents are declared identically – the only difference is which connection is passed. source is always derived from the connection's project_name.

Value

A list with exactly source, table, version, data_sha, and source_lineage. source is the parent connection's project_name; source_lineage is NULL when the snapshot carries none.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")

  # Resolve a parent declaration to pass to the parents argument of
  # datom_write.
  print(datom_parent(conn, "dm", datom_history(conn, "dm")$version[1]))

  unlink(tmp, recursive = TRUE)
}

List Projects Registered in the Governance Repo

Description

Returns a data frame with one row per project registered in the shared governance repo. Useful for managers and auditors who need to see the portfolio without having to clone every data repo.

Usage

datom_projects(x)

Arguments

Details

Accepts either a datom_conn (typically the developer's existing connection – reads the local gov clone) or a datom_store (lets a caller enumerate the portfolio before connecting to any specific project).

Read path:

• If a local gov clone is available (developer or any caller whose gov_local_path exists on disk), ⁠projects/⁠ is listed from disk and each ref.json is read locally. No network calls.

• Otherwise the gov storage client is used: ⁠projects/⁠ is listed and each projects/{name}/ref.json is fetched.

Corrupt registry entries (missing ref.json, unreadable JSON) emit a warning and are skipped – one bad project does not take down the listing.

Value

A data frame, sorted by name, with columns: name (character), data_backend (character), data_root (character), data_prefix (character; NA when absent), registered_at (character ISO8601 from clone mtime; NA on storage path).

Examples

# A governance store backed by a local directory. Projects are registered
# into it by the companion governance package (datomanager), so a freshly
# created governance store lists an empty portfolio.
tmp <- tempfile("datom-example-")
gov <- datom_store_local(file.path(tmp, "gov-storage"))

store <- datom_store(
  governance   = gov,
  data         = datom_store_local(file.path(tmp, "storage")),
  gov_repo_url = "https://github.com/example/acme-gov"
)

datom_projects(store)

unlink(tmp, recursive = TRUE)

Pull Latest Changes from Remote

Description

Fetches and merges the latest git changes from the remote repository. This is the recommended entry point at the start of each work session to ensure the local state is current before syncing or writing tables.

Usage

datom_pull(conn)

Arguments

Details

Git is the source of truth for all metadata (manifest, dispatch, table metadata). The manifest and other metadata files live in git and are pulled along with any other committed changes.

Requires developer role (readers have no git access).

Value

Invisibly, a list with:

commits_pulled

Integer count of new commits merged.

branch

Current branch name.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  # Nothing new on the remote yet, so this is a no-op.
  datom_pull(conn)

  unlink(tmp, recursive = TRUE)
}

Read a datom Table

Description

Unified read function with dispatch via dispatch.json. Reads from S3 metadata cache for data readers.

Usage

datom_read(conn, name, version = NULL, context = NULL, ...)

Arguments

Value

Data frame or routed function result.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")

  # Current version
  dm <- datom_read(conn, "dm")
  print(head(dm))

  # A specific version, by its identifier -- byte-for-byte the same table
  v <- datom_history(conn, "dm")$version[1]
  print(identical(datom_read(conn, "dm", version = v), dm))

  unlink(tmp, recursive = TRUE)
}

Write the Data-Side Governance Attachment Record

Description

Writes governance.json – the data-side pointer recording which governance repository a project is attached to. This is the data-repo / data-storage half of attaching governance; the gov-repo registration (writing ref.json and dispatch.json, committing to the gov repo) is performed separately by the governance layer (datomanager::gov_attach()).

Usage

datom_repo_attach_governance(conn, gov_repo_url, gov_store, message = NULL)

Arguments

Details

governance.json is the canonical data->gov pointer in the bidirectional governance link: the gov repo's ref.json points gov->data, and this file points data->gov, so either repo can find the other. It is written to two locations, mirroring the manifest pattern (git canonical, storage derived):

• .datom/governance.json in the local data clone (git canonical), committed and pushed to the data repo.

• {prefix}/datom/.metadata/governance.json in data storage (derived mirror; a failed mirror write warns but does not abort – the git copy is canonical and readers with gov access resolve location from the gov repo).

Routing this write through datom upholds the two-repos invariant: the governance layer never mutates the data repo directly.

Value

Invisibly, the SHA of the resulting data-repo commit.

See Also

datom_repo_delete(), datom_repo_set_data_store()

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  # Data-side half of governance attachment. The gov-repo registration
  # is performed separately by the companion datomanager package.
  datom_repo_attach_governance(
    conn,
    gov_repo_url = "https://github.com/example/acme-gov",
    gov_store    = datom_store_local(file.path(tmp, "gov-storage"))
  )
  gov_json <- jsonlite::read_json(
    file.path(tmp, "repo", ".datom", "governance.json")
  )
  print(gov_json$gov_repo_url)

  unlink(tmp, recursive = TRUE)
}

Delete the Data GitHub Repository and Local Clone

Description

Deletes the data-side GitHub repository via the GitHub REST API and removes the local clone directory. This is the data-side teardown step for a datom project.

Usage

datom_repo_delete(conn, confirm, force_gov_attached = FALSE)

Arguments

Details

Solo projects (no governance attached): call this together with datom_storage_delete_prefix() for a complete teardown.

Governed projects: use datomanager::gov_decommission() instead. That function calls datom_repo_delete() internally (with force_gov_attached = TRUE). Calling datom_repo_delete() directly on a governed project without that flag is refused to prevent accidentally orphaning the governance registration.

Steps:

1. Delete the data GitHub repo via the GitHub REST API (requires conn$github_pat with delete_repo scope; skipped with a warning when conn$github_pat is NULL or when the remote is not GitHub). Aborts if conn$data_repo_url is not set.

2. Remove the local clone directory (conn$path).

Each step is warn-and-continue on failure so the other still runs.

Value

Invisible TRUE on success.

See Also

datom_storage_delete_prefix(), datom_repo_set_data_store()

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage. Because the remote is not GitHub,
# the API deletion step is skipped and only the local clone is removed.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  # Solo project teardown (no governance): storage, then repo.
  datom_storage_delete_prefix(conn)
  datom_repo_delete(conn, confirm = conn$project_name)

  unlink(tmp, recursive = TRUE)
}

Rewrite the Data Store Pointer in project.yaml

Description

Updates storage.data in .datom/project.yaml to point at new_store, then commits and pushes the data repo. This is the data-side bookkeeping step of a store relocation.

Usage

datom_repo_set_data_store(conn, new_store, message = NULL)

Arguments

Details

Read-modify-write contract: the function reads the full existing project.yaml, modifies only storage.data, and writes back. It never reconstructs the file from conn fields. This preserves storage.governance on governed projects (it is permanent once written) and any other fields not owned by this function.

For governed projects the authoritative address is ref.json in the gov repo – this function updates only the local data clone so that datom_get_conn() stays consistent after migration. It is called by datomanager::gov_migrate_data() after the ref switch, never before.

Value

Invisibly, the SHA of the resulting commit.

See Also

datom_storage_copy(), datom_storage_verify(), datom_repo_delete()

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  # Repoint project.yaml at a relocated data store.
  new_store <- datom_store_local(file.path(tmp, "storage-relocated"))
  datom_repo_set_data_store(conn, new_store)

  unlink(tmp, recursive = TRUE)
}

Check datom Repository Structure

Description

Returns detailed check results for each component.

Usage

datom_repository_check(path)

Arguments

Value

List of TRUE/FALSE per check.

Show Repository Status

Description

Displays connection info, table count, and (for developers) uncommitted git changes and input file sync state.

Usage

datom_status(conn)

Arguments

Value

Invisibly, a list with connection, tables, and optionally git and input_files status details.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_status(conn)

  unlink(tmp, recursive = TRUE)
}

Copy All Objects Between Two datom Storage Namespaces

Description

Enumerates all objects under from_conn's datom namespace and streams each one to to_conn's datom namespace. All four backend combinations are supported:

Usage

datom_storage_copy(from_conn, to_conn)

Arguments

Details

• local -> local: direct file copy via fs::file_copy().

• local -> S3: reads raw bytes and uploads via put_object.

• S3 -> local: downloads via get_object and writes to disk.

• S3 -> S3: streams bytes through memory (get then put). Server-side copy_object (same-region optimisation) is reserved for a future release.

This is a policy-free primitive. It does not modify the source namespace, update project.yaml, or switch ref.json. For a complete managed migration (governed projects) use datomanager::gov_migrate_data(). For solo-project relocation combine this function with datom_repo_set_data_store().

Value

A data frame with columns key (character, relative key after ⁠{prefix}/datom/⁠) and bytes (numeric, byte count per object). Returns a zero-row data frame if the source namespace is empty.

See Also

datom_storage_verify(), datom_storage_list(), datom_storage_delete_prefix(), datom_repo_set_data_store()

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and two
# local directories for the source and destination object stores.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  from_conn <- datom_get_conn(file.path(tmp, "repo"), store)
  datom_write(from_conn, data = datom_example_data("dm"), name = "dm")

  # The destination is addressed with a reader connection: no local repo,
  # just a store plus the project name.
  to_store <- datom_store(data = datom_store_local(file.path(tmp, "storage2")))
  to_conn <- datom_get_conn(store = to_store, project_name = "example_project")

  copied <- datom_storage_copy(from_conn, to_conn)
  print(nrow(copied))      # number of objects copied
  print(sum(copied$bytes)) # total bytes

  unlink(tmp, recursive = TRUE)
}

Delete All Objects Under a datom Storage Prefix

Description

Removes every file under {prefix}/datom/{prefix_key} from storage. Pass prefix_key = NULL (the default) to delete the entire datom namespace for this connection. A missing or empty prefix is a no-op.

Usage

datom_storage_delete_prefix(conn, prefix_key = NULL)

Arguments

Details

Irreversible. Intended for package developers building tools on top of datom (e.g. datomanager for rollback or source deletion after migration). End users performing a full project teardown should use datom_repo_delete() instead.

Value

Invisibly, a backend-specific value. For S3: the count of deleted objects (0L if nothing found). For the local backend: 1L if the prefix directory existed and was removed, 0L otherwise.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)
  datom_write(conn, data = datom_example_data("dm"), name = "dm")

  # Delete a single table's objects
  datom_storage_delete_prefix(conn, prefix_key = "dm")

  # Delete the entire datom namespace (use with care)
  datom_storage_delete_prefix(conn)
  print(datom_storage_list(conn))

  unlink(tmp, recursive = TRUE)
}

List All Objects in a datom Storage Namespace

Description

Returns the full storage keys of every object under the datom namespace for this connection ({prefix}/datom/...). Intended for package developers building tools on top of datom (e.g. datomanager); end users typically do not need to inspect raw storage keys directly.

Usage

datom_storage_list(conn)

Arguments

Details

Keys are returned in their full storage-key form – for S3 that is "{prefix}/datom/..." relative to the bucket root; for local backends it is a path relative to conn$root. This mirrors the contract of the internal .datom_storage_list_objects() dispatch layer.

Value

A character vector of full storage keys. May be empty if the namespace contains no objects.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)
  datom_write(conn, data = datom_example_data("dm"), name = "dm")

  print(datom_storage_list(conn))

  unlink(tmp, recursive = TRUE)
}

Verify a Copy Between Two datom Storage Namespaces

Description

Checks that objects in to_conn's datom namespace match their counterparts in from_conn. Two verification modes are available:

Usage

datom_storage_verify(
  from_conn,
  to_conn,
  keys = NULL,
  mode = c("structural", "content")
)

Arguments

Details

• "structural" (default): Confirms each destination object exists and its byte size matches the source. Fast – one HEAD/stat per object, no byte transfer. Catches truncated or missing objects, which is the dominant copy failure mode.

• "content": Re-reads destination bytes, recomputes the SHA-256 hash, and compares against the source hash. Expensive (full re-download for remote backends) but gives true bit-level integrity. Use for regulated or paranoid runs.

Value

A data frame with columns:

• key (character): relative storage key.

• ok (logical): TRUE if the object passed verification.

• issue (character): description of the mismatch, or NA if ok. Returns a zero-row data frame if keys is empty.

See Also

datom_storage_copy(), datom_storage_list()

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and two
# local directories for the source and destination object stores.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  from_conn <- datom_get_conn(file.path(tmp, "repo"), store)
  datom_write(from_conn, data = datom_example_data("dm"), name = "dm")

  to_store <- datom_store(data = datom_store_local(file.path(tmp, "storage2")))
  to_conn <- datom_get_conn(store = to_store, project_name = "example_project")
  copied <- datom_storage_copy(from_conn, to_conn)

  # Verify all copied objects structurally (default, fast)
  results <- datom_storage_verify(from_conn, to_conn)
  print(all(results$ok))

  # Verify a subset with full content hash
  print(datom_storage_verify(from_conn, to_conn,
                             keys = copied$key[1],
                             mode = "content"))

  unlink(tmp, recursive = TRUE)
}

Create a datom Store

Description

Bundles a governance store component, a data store component, and git config into a single store object. Role (developer vs reader) is derived from github_pat presence.

Usage

datom_store(
  governance = NULL,
  data,
  github_pat = NULL,
  data_repo_url = NULL,
  gov_repo_url = NULL,
  gov_local_path = NULL,
  github_org = NULL,
  github_api_url = NULL,
  validate = TRUE
)

Arguments

Value

A datom_store object.

Examples

tmp <- tempfile("datom_store_")
store <- datom_store(
  data = datom_store_local(path = tmp),
  data_repo_url = "https://github.com/example/my-project",
  validate = FALSE
)
store
is_datom_store(store)
unlink(tmp, recursive = TRUE)

Create a Local Filesystem Store Component

Description

Constructs a validated local filesystem storage component for use as either the governance or data component of a datom_store. Validates that the path exists (or is creatable) and is writable.

Usage

datom_store_local(path, prefix = NULL, validate = TRUE)

Arguments

Value

A datom_store_local object.

Examples

tmp <- tempfile("datom_store_")
store <- datom_store_local(path = tmp, validate = TRUE)
store
is_datom_store_local(store)
unlink(tmp, recursive = TRUE)

Create an S3 Store Component

Description

Constructs a validated S3 storage component for use as either the governance or data component of a datom_store. Validates credentials and bucket access at construction time (unless validate = FALSE).

Usage

datom_store_s3(
  bucket,
  prefix = NULL,
  region = "us-east-1",
  access_key,
  secret_key,
  session_token = NULL,
  validate = TRUE
)

Arguments

Value

A datom_store_s3 object.

Examples

s3 <- datom_store_s3(
  bucket = "my-datom-bucket",
  prefix = "project/",
  region = "us-east-1",
  access_key = "AKIAIOSFODNN7EXAMPLE",
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  validate = FALSE
)
s3
is_datom_store_s3(s3)

Create a Credentials-Only S3 Store Component

Description

Constructs an S3 store component that carries only AWS credentials – no bucket, prefix, or region. The data location is resolved at connection time from ref.json stored in the governance repo. This is the recommended construction style for readers when a governance store is in place.

Usage

datom_store_s3_creds(access_key, secret_key, session_token = NULL)

Arguments

Details

A datom_store_s3_creds component must be paired with a governance component inside datom_store(). Attempting to create a composite store without governance will abort with a clear message.

Value

A datom_store_s3_creds object.

Examples

creds <- datom_store_s3_creds(
  access_key = "AKIAIOSFODNN7EXAMPLE",
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
)
creds
is_datom_store_s3_creds(creds)

Summarize a datom Project

Description

Returns a compact, role-aware overview of a datom project: its name, backend, table/version totals, last write time, and (for developers) the git remote URL. Reads .metadata/manifest.json from the data store.

Usage

datom_summary(conn)

Arguments

Value

A datom_summary S3 object (a list with class "datom_summary") containing: project_name, role, backend, root, prefix, table_count, total_versions, last_updated, remote_url. remote_url is NULL for readers (no local data clone).

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")
  print(datom_summary(conn))

  unlink(tmp, recursive = TRUE)
}

Sync Files to datom Repository

Description

Processes new/changed files from a manifest produced by datom_sync_manifest(). Imports each file via rio::import(), converts to a data frame, and calls datom_write() to store as parquet in S3 with git metadata. Updates the local .datom/manifest.json after each successful write.

Usage

datom_sync(conn, manifest, continue_on_error = TRUE)

Arguments

Value

The manifest data frame augmented with result and error columns. result is "success", "skipped", or "error".

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage. File import needs the optional
# rio package.
if (requireNamespace("git2r", quietly = TRUE) &&
    requireNamespace("rio", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  file.copy(
    system.file("extdata", "dm.csv", package = "datom"),
    file.path(tmp, "repo", "input_files", "dm.csv")
  )

  manifest <- datom_sync_manifest(conn)
  result <- datom_sync(conn, manifest)
  print(result[, c("name", "status", "result")])

  unlink(tmp, recursive = TRUE)
}

Scan and Prepare Manifest for Sync

Description

Scans a flat ⁠input_files/⁠ directory and computes file SHAs. Compares against the current .datom/manifest.json to detect new or changed files. Returns a manifest data frame for review before calling datom_sync().

Usage

datom_sync_manifest(conn, path = NULL, pattern = "*")

Arguments

Value

Data frame with columns: name, file, format, original_file_sha, status (one of "new", "changed", "unchanged", "unsupported_format").

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  # Drop a source file into the repo's input_files/ directory.
  file.copy(
    system.file("extdata", "dm.csv", package = "datom"),
    file.path(tmp, "repo", "input_files", "dm.csv")
  )

  manifest <- datom_sync_manifest(conn)
  print(manifest[, c("name", "format", "status")])

  unlink(tmp, recursive = TRUE)
}

Validate Git-Storage Consistency

Description

Checks that git metadata matches S3 storage for all tables and repo-level files. Reports mismatches as a structured result.

Usage

datom_validate(conn, fix = FALSE)

Arguments

Value

A list with:

valid

Logical — TRUE if everything is consistent.

repo_files

Data frame of repo-level file checks.

tables

Data frame of per-table checks.

fixed

Logical — TRUE if fix = TRUE was applied.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")
  datom_validate(conn)

  unlink(tmp, recursive = TRUE)
}

Write a datom Table

Description

Writes data to a datom repository. Commits to git, pushes, and syncs to S3.

Usage

datom_write(
  conn,
  data = NULL,
  name = NULL,
  metadata = NULL,
  message = NULL,
  parents = NULL,
  .source_lineage = NULL,
  .table_type = "derived",
  .original_file_sha = NULL,
  .original_format = NULL
)

Arguments

Value

List with deployment details.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  # --- Basic write (no lineage) ---
  dm <- datom_example_data("dm")
  datom_write(conn, data = dm, name = "dm")

  # --- Write with a single parent ---
  # Each parent's data_sha and lineage are resolved by datom_parent.
  lb <- datom_example_data("lb")
  datom_write(conn, data = lb, name = "lb")
  lb_summary <- aggregate(
    list(n = lb$LBTESTCD), by = list(LBTESTCD = lb$LBTESTCD), FUN = length
  )
  datom_write(
    conn,
    data    = lb_summary,
    name    = "lb_summary",
    message = "Lab test counts",
    parents = list(
      datom_parent(conn, "lb", datom_history(conn, "lb")$version[1])
    )
  )

  # --- Write with multiple parents ---
  # The source lineage is derived as the union of the parents' lineages.
  dm_lb_merged <- merge(dm, lb, by = "USUBJID")
  datom_write(
    conn,
    data    = dm_lb_merged,
    name    = "dm_lb_merged",
    message = "Demographics joined with lab results",
    parents = list(
      datom_parent(conn, "dm", datom_history(conn, "dm")$version[1]),
      datom_parent(conn, "lb", datom_history(conn, "lb")$version[1])
    )
  )

  print(datom_list(conn))

  unlink(tmp, recursive = TRUE)
}

Check if Object is a datom Connection

Description

Check if Object is a datom Connection

Usage

is_datom_conn(x)

Arguments

Value

TRUE or FALSE.

Check if Object is a datom Store

Description

Check if Object is a datom Store

Usage

is_datom_store(x)

Arguments

Value

TRUE or FALSE.

Examples

tmp <- tempfile("datom_store_")
store <- datom_store(
  data = datom_store_local(path = tmp),
  data_repo_url = "https://github.com/example/my-project",
  validate = FALSE
)
is_datom_store(store)
is_datom_store("not a store")
unlink(tmp, recursive = TRUE)

Check if Object is a Local Store Component

Description

Check if Object is a Local Store Component

Usage

is_datom_store_local(x)

Arguments

Value

TRUE or FALSE.

Examples

tmp <- tempfile("datom_store_")
store <- datom_store_local(path = tmp, validate = TRUE)
is_datom_store_local(store)
is_datom_store_local("not a store")
unlink(tmp, recursive = TRUE)

Check if Object is an S3 Store Component

Description

Check if Object is an S3 Store Component

Usage

is_datom_store_s3(x)

Arguments

Value

TRUE or FALSE.

Examples

s3 <- datom_store_s3(
  bucket = "my-datom-bucket",
  access_key = "AKIAIOSFODNN7EXAMPLE",
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  validate = FALSE
)
is_datom_store_s3(s3)
is_datom_store_s3("not a store")

Check if Object is a Credentials-Only S3 Store Component

Description

Check if Object is a Credentials-Only S3 Store Component

Usage

is_datom_store_s3_creds(x)

Arguments

Value

TRUE or FALSE.

Examples

creds <- datom_store_s3_creds(
  access_key = "AKIAIOSFODNN7EXAMPLE",
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
)
is_datom_store_s3_creds(creds)
is_datom_store_s3_creds("not a store")

Check if Path is a Valid datom Repository

Description

Validates datom repository structure. Used internally and by dpbuild.

Usage

is_valid_datom_repo(
  path,
  checks = c("all", "git", "datom", "renv"),
  verbose = FALSE
)

Arguments

Value

TRUE or FALSE.

Examples

# A plain directory is not a valid datom repository.
tmp <- tempfile("datom_valid_")
dir.create(tmp)
is_valid_datom_repo(tmp)
unlink(tmp, recursive = TRUE)

Create a datom Connection Object

Description

Internal constructor for the datom_conn S3 class. Two modes:

• Developer: has path to local repo + git access

• Reader: S3-only access, no local repo

Usage

new_datom_conn(
  project_name,
  root,
  prefix = NULL,
  region = "us-east-1",
  client,
  path = NULL,
  role = c("reader", "developer"),
  endpoint = NULL,
  gov_root = NULL,
  gov_prefix = NULL,
  gov_region = NULL,
  gov_backend = NULL,
  gov_client = NULL,
  gov_local_path = NULL,
  backend = "s3",
  data_repo_url = NULL,
  github_pat = NULL,
  github_api_url = NULL
)

Arguments

Details

The primary fields (root, prefix, region, client) refer to the data store. Governance store fields are prefixed with gov_.

Value

A datom_conn object.

Print a datom Connection

Description

Displays a clean summary without exposing credentials or the S3 client.

Usage

## S3 method for class 'datom_conn'
print(x, ...)

Arguments

Value

Invisible x.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  print(conn)

  unlink(tmp, recursive = TRUE)
}

Print a datom Store

Description

Displays store configuration with masked secrets.

Usage

## S3 method for class 'datom_store'
print(x, ...)

Arguments

Value

Invisible x.

Examples

tmp <- tempfile("datom_store_")
store <- datom_store(
  data = datom_store_local(path = tmp),
  data_repo_url = "https://github.com/example/my-project",
  validate = FALSE
)
print(store)
unlink(tmp, recursive = TRUE)

Print a Local Store Component

Description

Displays store configuration.

Usage

## S3 method for class 'datom_store_local'
print(x, ...)

Arguments

Value

Invisible x.

Examples

tmp <- tempfile("datom_store_")
store <- datom_store_local(path = tmp, validate = TRUE)
print(store)
unlink(tmp, recursive = TRUE)

Print an S3 Store Component

Description

Displays store configuration with masked secrets.

Usage

## S3 method for class 'datom_store_s3'
print(x, ...)

Arguments

Value

Invisible x.

Examples

s3 <- datom_store_s3(
  bucket = "my-datom-bucket",
  access_key = "AKIAIOSFODNN7EXAMPLE",
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  validate = FALSE
)
print(s3)

Print a Credentials-Only S3 Store Component

Description

Displays masked credentials and a note that location is resolved from ref.json at connection time.

Usage

## S3 method for class 'datom_store_s3_creds'
print(x, ...)

Arguments

Value

Invisible x.

Examples

creds <- datom_store_s3_creds(
  access_key = "AKIAIOSFODNN7EXAMPLE",
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
)
print(creds)

Print a datom_summary

Description

Print a datom_summary

Usage

## S3 method for class 'datom_summary'
print(x, ...)

Arguments

Value

Invisible x.

Examples

# Offline, self-contained: a bare git repo stands in for GitHub and a
# local directory for object storage.
if (requireNamespace("git2r", quietly = TRUE)) {
  tmp <- tempfile("datom-example-")
  remote <- file.path(tmp, "remote.git")
  dir.create(remote, recursive = TRUE)
  git2r::init(remote, bare = TRUE)

  store <- datom_store(
    data = datom_store_local(file.path(tmp, "storage")),
    github_pat = "example-token", # role selector; a local remote needs none
    data_repo_url = remote,
    validate = FALSE
  )
  datom_init_repo(file.path(tmp, "repo"), "example_project", store)
  conn <- datom_get_conn(file.path(tmp, "repo"), store)

  datom_write(conn, data = datom_example_data("dm"), name = "dm")
  print(datom_summary(conn))

  unlink(tmp, recursive = TRUE)
}