Version SHAs: The Three-SHA Identity Model

Why not just hash the parquet bytes

An earlier design hashed the serialized parquet file. That is simpler, and wrong for a versioning tool: parquet bytes move with the writer, not the data. An arrow upgrade, a different compression default, or a different dictionary-encoding heuristic all produce different bytes for identical content – and every one of those would have minted a spurious new version.

Hashing values instead of bytes decouples identity from the serializer. The serializer’s bytes are still hashed, but under a different name and for a different purpose: that is parquet_sha, below.

Where this bites

Most clinical and analytical tables are already inside the contract. The cases that are not tend to arrive the same handful of ways:

• A tidyr::nest() result carries a list column of data frames.
• strptime() returns POSIXlt, not POSIXct (which is a list under the hood, and hashing a list of broken-down time parts is not the same question as hashing an instant).
• sf geometry columns are sfc lists; units columns wrap a numeric with a unit attribute.
• A blob column from a database round-trip is a list of raw vectors.

None of these are errors in your data. They are values whose canonical byte form is a modelling decision, and datom asks you to make it explicitly rather than picking one silently.

Check before you write

datom_check_hashable() is the pre-flight entry point. It is pure – no connection, no store, no network – so you can run it on any data frame:

library(datom)

clean <- data.frame(
  id    = 1:3,
  score = c(1.5, 2.5, 3.5),
  grp   = factor(c("x", "y", "x")),
  day   = as.Date(c("2026-01-01", "2026-01-02", "2026-01-03"))
)
datom_check_hashable(clean)
#> ✔ All 4 columns are hashable. This table is ready for `datom_write()`.

messy <- data.frame(id = 1:2)
messy$notes <- list(c("a", "b"), "c")
messy$z <- c(1 + 2i, 3 + 4i)

report <- datom_check_hashable(messy)
#> ✖ 2 of 3 columns are not hashable.
#> ✖ Column notes (<list>): List and blob columns are not hashable. Flatten to one
#>   value per row with tidyr::unnest(), or serialize each element to character
#>   (for example with jsonlite::toJSON() per element), before writing.
#> ✖ Column z (<complex>): Complex columns are not hashable. Split into separate
#>   real and imaginary numeric columns, or convert to character, before writing.
#> ℹ Fix these before `datom_write()`, which refuses the whole table until they are resolved.
report[report$status == "unsupported", c("column", "class")]
#>   column   class
#> 2  notes    list
#> 3      z complex

The report is returned invisibly as a data frame, so it composes: filter it, kable() it, or assert on it in a pipeline.

datom_write() runs the same check itself, before it touches git or storage, and reports every offending column in one error rather than one per attempt. A refusal leaves no partial state behind.

The recourse table

Each refused type has one canonical piece of advice. The table below is generated by running datom_check_hashable() over one fixture per case, so it is the same text datom_write() aborts with – the documentation cannot drift from the code.

The ingestion allowlist

datom_sync() onboards flat tabular files only: csv, tsv, txt, psv, parquet, sas7bdat, xpt, sav, zsav, por, dta, xls, xlsx. Anything else – .rds, .json, .xml – is refused up front. datom_sync_manifest() flags it as unsupported_format and still processes its allowlisted siblings; datom_sync() reports the same recourse in the error column for that row.

The reason is the table contract: a container format can deserialize to anything, including the list and exotic columns the hash refuses. Refusing at the door keeps the failure legible (“this format is not onboarded”) instead of deferring it to a type error deep in the write.

The escape hatch, and its tradeoff. You can always read an unsupported source yourself and pass the resulting data frame to datom_write():

dm <- readRDS("dm.rds")
datom_write(conn, data = dm, name = "dm")

The rule to know: that table is derived, not imported. It has no original_file_sha, so input-file change detection does not apply to it. A re-run will not report “unchanged” from the file’s bytes; it compares the data. If you need file-level change detection for that source, convert it to CSV or parquet once and let datom_sync() onboard it.

The dedup guard is full-history

When a version’s metadata_sha already appears anywhere in version_history.json, no entry is appended – the current pointer in metadata.json is still updated. This matters when you re-sync an older file that content-matches: without a full-history scan, that write would append a second entry with a version value already present, and a later datom_read(version = ...) would report the SHA as ambiguous.

One caveat that is not ours to fix

data_sha is a portable identity of doubles. Getting from text to doubles is a separate step, and in base R it is neither correctly rounded nor platform-independent. R does its own decimal parsing (R_strtod in src/main/util.c): it accumulates digits into a long double and then scales by a power of ten. So the result depends on how wide long double is on the build – and as.numeric(), decimal literals, scan() and read.csv() all go through it.

This is known, accepted R behaviour rather than a bug: CRAN runs a dedicated check flavour with R configured --disable-long-double precisely because results differ, and package authors are told to tolerate it.

Check your own platform in one line:

.Machine$sizeof.longdouble   # 8 means long double == double
#> [1] 8

8 means no extra precision, and decimal parsing on that build deviates from the correctly-rounded value. It is not an Apple-versus-Intel split:

Note that Linux and macOS on the same arm64 chip family fall on opposite sides – the variable is the long double width, not the architecture.

When it bites. Deviation is possible whenever mantissa * 10^exponent cannot be reached by a single correctly-rounded operation at the platform’s long double precision – that is, when the accumulated mantissa exceeds the significand, or 10^|exponent| is not itself exactly representable. 10^22 is the largest power of ten exactly representable as a double, and 10^23 the first that is not, which is why on Apple silicon a bare 1e-23 is the first one-digit value to drift. Longer mantissas drift sooner: 3.14159265358979e-9 is already 1 ULP off, and 1e300 / 1e-300 are 4. Short decimals – 0.1, 1, 1e15, 3.14159265358979 – are exact and agree everywhere, because only a single rounding occurs.

The consequence for datom. Two machines whose builds fall on opposite sides of that table can parse different doubles from the same CSV text, and will then compute different data_sha values – both correct, for genuinely different tables. No hash can paper over it; the divergence happens before datom sees the data.

Mitigations, in order of preference:

• Read via arrow, or store as parquet. Arrow’s CSV reader uses fast_float, which is correctly rounded and platform-independent by design; parquet stores the doubles themselves, so no decimal parsing happens at all. This is the recommendation.
• Keep the ingesting platform fixed for a given table if you must go through base-R CSV reading.
• For exact literals in code, use C99 hex-float notation – 0x1.999999999999ap-4 is exactly 0.1, and 0x1.fcp+996 a large value with no base-10 rounding anywhere. Documented in ?NumericConstants and portable.

One warning: data.table::fread() is not a fix. It has its own parser that is also not correctly rounded, and its errors differ from base R’s – the two can disagree on the same string.

Worth keeping in perspective: this is a property of the platform’s decimal parser, not of datom, and it affects any analysis reading that CSV with or without datom. What datom adds is that the difference becomes visible instead of silent.