---
title: "Version SHAs: The Three-SHA Identity Model"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Version SHAs: The Three-SHA Identity Model}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment  = "#>",
  eval     = FALSE
)
```

> **Companion to**: [Getting Started](getting-started.html). Read this when
> you've noticed that `datom_history()` shows a `version` column and a
> `data_sha` column and want to know why there is more than one hash -- or
> when `datom_write()` has told you a column is not hashable.

datom identifies a table with **three SHAs**, each answering a different
question:

| SHA | Question it answers | Computed over |
|---|---|---|
| `data_sha` | *Is this the same table content?* | the table's values, canonically encoded (`datom-cv1`) |
| `metadata_sha` | *Is this the same version?* | the table's semantic metadata, as canonical JSON |
| `parquet_sha` | *Are the stored bytes intact?* | the exact parquet object that was uploaded |

A fourth hash, `original_file_sha`, is recorded for tables onboarded from a
file by `datom_sync()`. It is not part of table identity -- it is *file*
provenance, and it is what lets a re-scan say "this input has not changed"
without reading the file.

The rest of this article explains each one, the type contract `data_sha`
depends on, and the identity decisions baked into it.

## `data_sha`: the canonical content hash

`data_sha` is a hash of the table's **values**, computed with the `datom-cv1`
algorithm. Sketching the layout:

```r
# per column, in table column order
col_digest <- sha256(utf8(type_tag) || utf8(column_name) || 0x00 || payload)

# then the table
data_sha <- sha256("datom-cv1" || f64le(nrow) || f64le(ncol) ||
                   concat(col_digest_hex ...))
```

Two properties matter more than the layout:

- **No I/O and no parquet.** `data_sha` is computed from the in-memory data
  frame. It never writes a file, never calls arrow, and never coerces the
  container.
- **The container is not identity.** A `tibble`, a `data.table`, and a
  `data.frame` holding the same values hash the same. Attributes that are not
  values (row names, `tbl_df` class, arbitrary `attr()`s) do not participate.

### Why not just hash the parquet bytes

An earlier design hashed the serialized parquet file. That is simpler, and
wrong for a versioning tool: parquet bytes move with the **writer**, not the
data. An arrow upgrade, a different compression default, or a different
dictionary-encoding heuristic all produce different bytes for identical
content -- and every one of those would have minted a spurious new version.

Hashing values instead of bytes decouples identity from the serializer. The
serializer's bytes are still hashed, but under a different name and for a
different purpose: that is `parquet_sha`, below.

## The datom table contract

Hashing values means datom must decide, per column type, *what the bytes of
that value are*. That decision cannot be guessed, so `datom-cv1` supports a
defined set of types and refuses everything else with specific advice.

Supported: `logical`, `integer`, `double`, `character`, `factor`, `Date`
(including `data.table::IDate`), `POSIXct`, `difftime`/`hms`,
`data.table::ITime`, `bit64::integer64`, and labelled vectors
(`haven_labelled`, `labelled`, `labelled_spss`) over any of those.

### Where this bites

Most clinical and analytical tables are already inside the contract. The
cases that are not tend to arrive the same handful of ways:

- **A `tidyr::nest()` result** carries a list column of data frames.
- **`strptime()`** returns `POSIXlt`, not `POSIXct` (which is a list under the
  hood, and hashing a list of broken-down time parts is not the same question
  as hashing an instant).
- **`sf`** geometry columns are `sfc` lists; **`units`** columns wrap a
  numeric with a unit attribute.
- **A blob column** from a database round-trip is a list of raw vectors.

None of these are errors in your data. They are values whose canonical byte
form is a modelling decision, and datom asks you to make it explicitly rather
than picking one silently.

### Check before you write

`datom_check_hashable()` is the pre-flight entry point. It is pure -- no
connection, no store, no network -- so you can run it on any data frame:

```{r, eval = TRUE}
library(datom)

clean <- data.frame(
  id    = 1:3,
  score = c(1.5, 2.5, 3.5),
  grp   = factor(c("x", "y", "x")),
  day   = as.Date(c("2026-01-01", "2026-01-02", "2026-01-03"))
)
datom_check_hashable(clean)
```

```{r, eval = TRUE}
messy <- data.frame(id = 1:2)
messy$notes <- list(c("a", "b"), "c")
messy$z <- c(1 + 2i, 3 + 4i)

report <- datom_check_hashable(messy)
report[report$status == "unsupported", c("column", "class")]
```

The report is returned invisibly as a data frame, so it composes: filter it,
`kable()` it, or assert on it in a pipeline.

`datom_write()` runs the same check itself, **before** it touches git or
storage, and reports every offending column in one error rather than one per
attempt. A refusal leaves no partial state behind.

### The recourse table

Each refused type has one canonical piece of advice. The table below is
generated by running `datom_check_hashable()` over one fixture per case, so it
is the same text `datom_write()` aborts with -- the documentation cannot drift
from the code.

```{r, eval = TRUE, echo = FALSE}
one_col <- function(value) {
  d <- data.frame(placeholder = rep(NA, max(1L, length(value))))
  d$col <- value
  d["placeholder"] <- NULL
  d
}

# One fixture per recourse case. datom dispatches on the class tag, so a bare
# structure() carrying the class is enough to render the advice for the
# package-specific cases without taking a dependency on sf / units / zoo.
fixtures <- list(
  as.POSIXlt("2026-01-01", tz = "UTC"),
  structure(list(1, 2), class = "sfc"),
  list(data.frame(a = 1), data.frame(a = 2)),
  list(1:2, letters[1:2]),
  structure(c(1, 2), class = "units"),
  structure(c(2026.0, 2026.1), class = "yearmon"),
  c(1 + 2i, 3 + 4i),
  as.raw(1:2),
  structure(1:2, class = "myclass")
)

report <- do.call(rbind, lapply(fixtures, function(v) {
  suppressMessages(datom_check_hashable(one_col(v)))
}))

knitr::kable(
  data.frame(Column_class = report$class, Recourse = report$recourse),
  col.names = c("Column class", "Recourse")
)
```

### The ingestion allowlist

`datom_sync()` onboards **flat tabular files only**: `csv`, `tsv`, `txt`,
`psv`, `parquet`, `sas7bdat`, `xpt`, `sav`, `zsav`, `por`, `dta`, `xls`,
`xlsx`. Anything else -- `.rds`, `.json`, `.xml` -- is refused up front.
`datom_sync_manifest()` flags it as `unsupported_format` and still processes
its allowlisted siblings; `datom_sync()` reports the same recourse in the
`error` column for that row.

The reason is the table contract: a container format can deserialize to
anything, including the list and exotic columns the hash refuses. Refusing at
the door keeps the failure legible ("this format is not onboarded") instead of
deferring it to a type error deep in the write.

**The escape hatch, and its tradeoff.** You can always read an unsupported
source yourself and pass the resulting data frame to `datom_write()`:

```{r}
dm <- readRDS("dm.rds")
datom_write(conn, data = dm, name = "dm")
```

The rule to know: that table is **derived**, not imported. It has no
`original_file_sha`, so input-file change detection does not apply to it. A
re-run will not report "unchanged" from the file's bytes; it compares the
data. If you need file-level change detection for that source, convert it to
CSV or parquet once and let `datom_sync()` onboard it.

## The identity decisions

Every entry below is a deliberate choice about what counts as "the same
table".

| Decision | Rationale |
|---|---|
| `logical`, `integer`, `double` unify (same encoding, same tag) | `1L` and `1` are the same number. Round-tripping a table through CSV, parquet, or a database routinely flips storage type without changing the data. |
| Factor **levels** and **orderedness** are not identity | A factor's values are hashed as character. Re-leveling for a plot, or dropping unused levels, must not mint a new data version. |
| Value **labels** are not identity | `haven_labelled` columns strip to their underlying type. A relabelled SAS import carries the same numbers. |
| `POSIXct` **tzone** is not identity | The value hashed is the instant (epoch seconds). `tzone` is a display attribute; the same instant printed in two zones is one instant. |
| All `NaN` payloads collapse to one canonical `NaN`; `-0` encodes as `+0` | `0/0` and `NaN` are indistinguishable arithmetically, and IEEE-754 leaves NaN payload bits and signed zero to the platform. Letting them differ would make identity platform-dependent. |
| `NA_real_` stays distinct from `NaN` | "Missing" and "not a number" are different statements about the data, and R keeps them apart. |
| Doubles are hashed **bit-exact** -- no rounding, no tolerance | A tolerance would make identity non-transitive (a ~ b and b ~ c but a != c). If a pipeline produces last-bit differences, that is a fact about the pipeline; round explicitly if you want it hidden. |
| Unicode NFC and NFD forms differ | Normalizing would require an ICU dependency and a normalization-form decision baked into every future version. Byte-identical text is the narrow, stable promise. |
| Row order and column order are **significant**; nothing is sorted | Sorting to make identity order-independent needs a collation order, which is locale-dependent -- reintroducing the platform dependence the rest of this design removes. A reordered table is a different table; `arrange()` deliberately if order should not matter. |

**On the apparent asymmetry**: `NaN` payloads are canonicalized while NFD text
is not, and storage types unify while row order does not. The line is drawn at
**platform non-determinism**, not at "things a human would call equivalent".
Where the same logical value can have different bytes for reasons outside the
user's control (NaN payload bits, signed zero, integer-vs-double storage after
a round-trip), datom canonicalizes -- otherwise identity would depend on the
machine. Where a difference is a real, visible, user-controlled property of
the data (row order, NFC vs NFD, secs vs mins), datom preserves it: collapsing
those would need a policy decision that belongs to the user, and any policy
datom picked would be wrong for someone.

## `metadata_sha`: the version

`metadata_sha` is what `datom_history()` calls `version` and what you pass to
`datom_read(version = ...)`. It hashes the table's semantic metadata:

```r
volatile  <- c("created_at", "datom_version", "parquet_sha",
               "column_hashes", "size_bytes")
semantic  <- metadata[setdiff(names(metadata), volatile)]
sorted    <- semantic[sort(names(semantic), method = "radix")]
canonical <- jsonlite::toJSON(sorted, auto_unbox = TRUE)
digest::digest(canonical, algo = "sha256", serialize = FALSE)
```

Three details are load-bearing:

1. **The JSON canonical form is hashed, not the R object.** R's `serialize()`
   is type-sensitive (`10L` and `10` differ); JSON-parsed metadata has lost
   that distinction. A reader pulling metadata from storage must compute the
   same `metadata_sha` as the writer who built it in memory, so the shared
   representation is the JSON text. `serialize = FALSE` is what makes
   `digest()` hash those bytes.
2. **`method = "radix"` sorts field names by byte value.** Plain `sort()` is
   locale-collated, so two machines with different `LC_COLLATE` would order
   fields differently and produce different versions for identical metadata.
3. **The volatile set is excluded on purpose.** `created_at` and
   `datom_version` are wall-clock and package facts. `parquet_sha` and
   `size_bytes` move with the arrow version for identical content -- letting
   them in would undo the whole point of hashing values. `column_hashes` is a
   deterministic function of the same values that produce `data_sha`, so it
   carries no independent information.

Conversely, `original_file_sha` and `hash_algo` **are** semantic: a new source
file is a new version of the table's provenance, and a new hash algorithm is a
new identity regime.

### The dedup guard is full-history

When a version's `metadata_sha` already appears **anywhere** in
`version_history.json`, no entry is appended -- the current pointer in
`metadata.json` is still updated. This matters when you re-sync an older file
that content-matches: without a full-history scan, that write would append a
second entry with a `version` value already present, and a later
`datom_read(version = ...)` would report the SHA as ambiguous.

## `parquet_sha`: stored-object integrity

`parquet_sha` is the SHA-256 of the exact parquet object in storage. It is not
identity -- it is a tamper and corruption check:

- On read, the downloaded object is hashed and compared **before** parsing.
  A mismatch aborts with the key, the expected hash, and the actual one; it
  never hands you data it cannot vouch for.
- When a write is `metadata_only` (the content is unchanged), the current
  `parquet_sha` is **carried forward** and nothing is uploaded.
- When a write reverts to content already in history, the recorded
  `parquet_sha` is **reused** and the stored object is left alone. Re-serializing
  can differ byte-for-byte, which would break the older version's integrity
  pin for no benefit.
- Metadata written before this hash existed has no `parquet_sha`, and the check
  is **skipped** rather than failed -- a read-time grace, not a migration.

## Change classification

Every `datom_write()` (including the one inside `datom_sync()`) lands in
exactly one of three outcomes:

| Outcome | When | Effect |
|---|---|---|
| `none` | `data_sha` and `metadata_sha` both unchanged | no commit, no upload, no history entry |
| `metadata_only` | same `data_sha`, different `metadata_sha` (e.g. a re-export with new `original_file_sha`, or new parents) | new version, `parquet_sha` carried forward, **no upload** |
| `full` | `data_sha` changed | new version, parquet stored at `{name}/{data_sha}.parquet` |

The `metadata_only` case is the one that pays for the whole design: an EDC
system re-exporting the same data with a new timestamp in the header produces
different file bytes and identical content. datom records the new provenance
and stores nothing twice.

## The column index

Alongside `data_sha`, metadata carries `column_hashes`: an ordered array of
`{name, sha}`, one entry per column, in table column order, untruncated. It is
the same per-column digest that fed `data_sha`, computed once and reused.

Two things follow. `data_sha` can be re-derived from the index plus the
dimensions without downloading any data, which makes it cheaply self-checkable.
And two versions that differ in exactly one column differ in exactly one
entry -- the anchor a future column-level diff builds on.

## How far the cross-language claim goes

- **Within R, pinned with `renv`: guaranteed.** The goldens in the test suite
  pin the byte layout across the supported platform matrix.
- **The specification is language-implementable.** `datom-cv1` is defined in
  terms of bytes -- UTF-8 tags, little-endian IEEE-754 doubles, NUL-terminated
  strings, SHA-256 -- not R semantics. `dev/datom_cv1_reference.R` is a
  standalone base-R + `digest` reference implementation that the package is
  tested byte-for-byte against, and it doubles as the spec any other language
  would implement. What is **not** claimed is that an arbitrary Python or Julia
  reader will agree by accident; agreement requires implementing the same
  encodings, starting with that reference.
- **Raw-file onboarding identity is language-independent today.**
  `original_file_sha` is a SHA-256 of the file's bytes, so any tool in any
  language computes it identically. Change detection on inputs therefore does
  not depend on the hashing algorithm at all.

### One caveat that is not ours to fix

`data_sha` is a portable identity of *doubles*. Getting from **text to
doubles** is a separate step, and in base R it is neither correctly rounded nor
platform-independent. R does its own decimal parsing (`R_strtod` in
`src/main/util.c`): it accumulates digits into a `long double` and then scales
by a power of ten. So the result depends on how wide `long double` is on the
build -- and `as.numeric()`, decimal literals, `scan()` and `read.csv()` all go
through it.

This is known, accepted R behaviour rather than a bug: CRAN runs a dedicated
check flavour with R configured `--disable-long-double` precisely because
results differ, and package authors are told to tolerate it.

Check your own platform in one line:

```{r, eval = TRUE}
.Machine$sizeof.longdouble   # 8 means long double == double
```

`8` means no extra precision, and decimal parsing on that build deviates from
the correctly-rounded value. It is **not** an Apple-versus-Intel split:

| Build | `long double` | Decimal parsing |
|---|---|---|
| x86_64 (Linux, Windows, Intel mac) | 80-bit extended | agrees with correct rounding on everything tested |
| aarch64 **Linux**, s390x, riscv64 | 128-bit quad | agrees |
| **Apple silicon macOS**, 32-bit ARM | same as `double` | **deviates** |
| any build with `--disable-long-double` (CRAN's noLD flavour) | forced to `double` | **deviates** |

Note that Linux and macOS on the *same* arm64 chip family fall on opposite
sides -- the variable is the `long double` width, not the architecture.

**When it bites.** Deviation is possible whenever `mantissa * 10^exponent`
cannot be reached by a single correctly-rounded operation at the platform's
`long double` precision -- that is, when the accumulated mantissa exceeds the
significand, or `10^|exponent|` is not itself exactly representable. `10^22` is
the largest power of ten exactly representable as a double, and `10^23` the
first that is not, which is why on Apple silicon a bare `1e-23` is the first
one-digit value to drift. Longer mantissas drift sooner: `3.14159265358979e-9`
is already 1 ULP off, and `1e300` / `1e-300` are 4. Short decimals -- `0.1`,
`1`, `1e15`, `3.14159265358979` -- are exact and agree everywhere, because only
a single rounding occurs.

**The consequence for datom.** Two machines whose builds fall on opposite sides
of that table can parse *different doubles* from the same CSV text, and will
then compute different `data_sha` values -- both correct, for genuinely
different tables. No hash can paper over it; the divergence happens before
datom sees the data.

Mitigations, in order of preference:

- **Read via `arrow`, or store as parquet.** Arrow's CSV reader uses
  `fast_float`, which is correctly rounded and platform-independent by design;
  parquet stores the doubles themselves, so no decimal parsing happens at all.
  This is the recommendation.
- **Keep the ingesting platform fixed** for a given table if you must go
  through base-R CSV reading.
- **For exact literals in code**, use C99 hex-float notation --
  `0x1.999999999999ap-4` is exactly `0.1`, and `0x1.fcp+996` a large value with
  no base-10 rounding anywhere. Documented in `?NumericConstants` and portable.

One warning: `data.table::fread()` is **not** a fix. It has its own parser that
is also not correctly rounded, and its errors differ from base R's -- the two
can disagree on the same string.

Worth keeping in perspective: this is a property of the platform's decimal
parser, not of datom, and it affects any analysis reading that CSV with or
without datom. What datom adds is that the difference becomes *visible* instead
of silent.

## What you can rely on

- A version is uniquely identified by its `metadata_sha`, and that SHA never
  appears twice in a table's history.
- The same content under different provenance shares one `data_sha` and one
  stored parquet object, with a distinct `metadata_sha` per version.
- Identical writes are no-ops at every layer: no commit, no upload, no history
  entry.
- A reader on another machine, in another locale, computes the same
  `metadata_sha` as the writer.
- Data you read back is the data that was written, or the read fails loudly.