Package {SSPICE}

SSPICE: Systematic Skip-Pattern Integrity and Consistency Engine

Description

A graph-based framework for representing, analyzing, validating, and correcting structured statistical data governed by explicit logical dependencies.

Details

SSPICE stands for Systematic Skip-Pattern Integrity and Consistency Engine. Start with the overview vignette and the repository README. The framework is described in Massoli (2026), An Integrated Framework for Rule-Based Data Editing, https://arxiv.org/abs/2610.16947.

Author(s)

Maintainer: Pierpaolo Massoli pimassol@istat.it

Build and analyze a directed acyclic graph

Description

Builds the SSPICE graph representation from routing metadata. Logical OR and XOR groups are coarsened before constructing the transposed incidence matrix and the adjacency matrix. No record-level data are required.

Usage

build_DAG(
  routing,
  separator = ";",
  na_strings = c("", ".", "NA", "NULL"),
  analyze = TRUE,
  stop_on_cycle = TRUE,
  max_paths_per_partition = 1e+06,
  start_node = 1L,
  end_node = NULL
)

Arguments

Value

An object of class sspice_dag. It contains the original and transformed routing metadata, node names, the transposed incidence matrix Bt, the adjacency matrix A, and optional node-level and graph-level diagnostics.

Examples

routing <- data.frame(
  V1 = c("Q1", "Q2"),
  V2 = c(1, 1),
  V3 = c(1, 1),
  V4 = c(1, 1),
  V5 = c(0, 0),
  V6 = c(0, 0)
)
dag <- build_DAG(routing)
dag$Bt
dag$A

Inspect or build a cached routing-check artifact

Description

The cache is reused only when its manifest matches the current input files, package version, cache schema and routing-check parameters. Input files are fingerprinted with MD5 checksums.

Usage

cached_routing_check(
  data,
  dag,
  record_id,
  logical_edit_names = NULL,
  items_mapping_file = NULL,
  cache_file,
  input_files,
  cache_policy = c("auto", "refresh", "readonly", "off"),
  check_function = NULL
)

Arguments

Value

A list containing result, manifest, and cache diagnostics.

Calculate HITS scores

Description

Calculates hub and authority scores for the nodes of a directed graph by applying the HITS iteration to its adjacency matrix.

Usage

calculate_hits(A, max_iter = 100L, tol = 1e-06)

Arguments

Value

A list containing authority, hub, iterations, and converged.

Detect all violated logical-edit variables

Description

Internal helper that reports every missing logical-edit value independently of the path followed through the questionnaire DAG.

Usage

check_logical_edits(data, id_col, logical_edit_names)

Arguments

Value

A report compatible with the full report returned by routing_check().

Correct questionnaire records using MCMC simulated annealing

Description

Correction is restricted to the local neighbourhood of reported routing or logical-edit violations. Every fill or flip samples a mode from the node domain. Immutable donor matrices estimate the conditional distribution of each variable given its immediate predecessor and strata, including when the predecessor is missing, with predecessor-only and marginal fallbacks. Complete donor records are never copied or updated with corrected records.

Usage

correct_data(
  data,
  dag,
  record_id = NULL,
  donor_data = NULL,
  donor_data_validated = FALSE,
  donor_cache = NULL,
  vaux_metadata = NULL,
  logical_edit_metadata = NULL,
  immutable_variables = NULL,
  strata = NULL,
  imputation_method = c("donor", "user_prior", "uniform"),
  imputation_priors = NULL,
  domains = NULL,
  external_data_dir = ".",
  items_mapping_file = NULL,
  control = correction_control(),
  batch_output_dir = NULL
)

Arguments

Details

corrected_data contains the supplied rows after correction in the prepared representation. record_summary contains per-record diagnostics and resolved; unresolved_records is its unresolved subset. final_routing_check contains the final violation reports. donor_diagnostics gives supplied and retained donor counts. control, strata and immutable_variables record settings; parallel and batch_manifest describe execution and checkpoints.

Random generation uses deterministic per-record seeds. The caller's random state and RNG kind are restored on exit, including on errors. Results do not depend on worker count or batch boundaries. Random sequences differ from the initial CRAN submission; checkpoints from that submission cannot be reused.

The operational Step 10, rather than this function, performs the initial EXACT/ERRONEOUS partition and pilot selection. A finite stochastic search does not guarantee a feasible correction or a global minimum-change result. See the overview vignette for a complete standalone example.

Value

An object of class sspice_correction_result.

Territorial strata in the example

In the supplied example, STRATUM identifies five geographical divisions of Italy: North-West (1), North-East (2), Centre (3), South (4), and Islands (5). It is used for stratification during correction. With freeze_strata = TRUE, stratum membership remains unchanged.

Preparation before the first batch

Before record correction begins, SSPICE prepares or reuses conditional distributions, donor information, variable domains and other supporting structures. Parallel execution also requires worker initialization. Step 10 performs an initial routing check unless a valid cached result is available.

Preparation time depends on the number of variables and records, the number of categories and strata, and the routing structure. Computing conditional distributions can take appreciable time before the first batch starts.

The first batch-completion message appears only after preparation and processing of that batch. A delay before this message does not by itself indicate that execution has stalled.

Distinguish total script elapsed time from the correction time reported in the output. Their difference is not a complete measure of initialization time, because some preparation may occur within the timed correction call. Measuring individual preparation stages requires separate timing.

Examples

local({
  routes <- read.table(text = paste(
    "ID;1;1;9999;0;0;.;.",
    "GATE;2;1;2;0;1;1;END",
    "DETAIL;2;1;2;0;0;.;.",
    "END;1;1;1;0;0;.;.", sep = "\n"),
    sep = ";", header = FALSE, na.strings = ".")
  dag <- SSPICE::build_DAG(routes, analyze = FALSE)
  raw <- data.frame(ID = 1:3, GATE = c(1, 2, 1),
                    DETAIL = c(NA, 1, 1), END = c(1, 1, 1))
  prepared <- SSPICE::prepare_data(raw, dag)

  # The current checker requires a mapping file, even with no recoding.
  mapping <- tempfile(fileext = ".csv")
  on.exit(unlink(mapping), add = TRUE)
  write.csv(data.frame(varname = character(), item_route = numeric(),
                       item_dataset = numeric()), mapping, row.names = FALSE)
  check <- SSPICE::routing_check(
    prepared$data, dag$routes_numeric, dag$Bt, "ID",
    basename(mapping), dirname(mapping)
  )
  print(check$full_report)
  stopifnot(setequal(unique(check$full_report$ID), 3))

  # Only the erroneous row is sent to correction. Donors are fixed.
  result <- SSPICE::correct_data(
    data = prepared$data[3, , drop = FALSE], dag = dag, record_id = "ID",
    donor_data = prepared$data[1:2, , drop = FALSE],
    items_mapping_file = mapping,
    control = SSPICE::correction_control(
      seed = 123L, max_iterations = 20L, use_parallel = FALSE
    )
  )
  print(result$corrected_data)
  print(result$record_summary)
  print(result$unresolved_records)
  stopifnot(identical(result$corrected_data$ID, prepared$data$ID[3]))
  # Resolution is read from the final checks, not assumed from termination.
  print(result$final_routing_check$full_report)
})

Configure SSPICE data correction

Description

Configure SSPICE data correction

Usage

correction_control(
  routing_weight = 1,
  logical_edit_weight = 1,
  vaux_weight = 1,
  lambda_change = 2,
  future_weight = 0.5,
  penalty_fill = 1,
  penalty_blank = 2,
  penalty_flip = 3,
  initial_temperature = NULL,
  final_temperature = 1e-04,
  cooling_fraction = 0.9,
  max_iterations = NULL,
  top_k = 10L,
  beta_fill = 1,
  max_nodes_per_iteration = 1L,
  logical_repair_max_evaluations = 1000L,
  logical_repair_max_changes = 2L,
  logical_repair_beam_width = 10L,
  minimum_donors = 1L,
  allow_unconditional_donor = TRUE,
  branch_fill_threshold = 0.5,
  freeze_strata = TRUE,
  seed = 1234L,
  use_parallel = FALSE,
  parallel_workers = NULL,
  batch_size = NULL,
  resume_batches = FALSE
)

Arguments

Value

A list of class sspice_correction_control.

Examples

control <- correction_control(seed = 123L, use_parallel = FALSE)
control$max_iterations
control$final_temperature

Build a quality report for an SSPICE correction run

Description

Produces read-only diagnostics describing the input data, violations before and after correction, changed cells, correction invasiveness, execution characteristics, categorical post-hoc uncertainty, and presence/absence actions on quantitative variables. It never changes the correction result or any input record.

Usage

correction_quality_report(
  original_data,
  correction_result,
  dag,
  corrected_data = NULL,
  initial_check = NULL,
  final_check = NULL,
  donor_data = NULL,
  record_id = NULL,
  output_dir = NULL,
  elapsed_seconds = NA_real_,
  donor_origin = NA_character_
)

Arguments

Details

Variable measurement type is determined without heuristic interpretation: textual columns are STRING; non-textual routing nodes with nitems == 1 and minval < maxval are QUANTITATIVE; all other variables are QUALITATIVE.

The result contains summary (metric/value pairs), variable_profile, record_summary, unresolved_records, violations_before, violations_after, changes, their record/variable summaries, uncertainty and quantitative_structural_changes.

The editable-cell denominator is the number of distinct processed IDs times the number of non-ID columns not marked immutable. It is not limited to active questions or base variables. Changed-cell percentage is 100 times the number of changes divided by this denominator, or NA if it is zero.

Categorical entropy uses natural logarithms; normalized entropy divides by the logarithm of positive support size (zero for support size at most one). These are post-hoc distributions at the final record, not confidence intervals or a replay of sampled proposals. User priors and domain overrides are not restored by this reconstruction. Inspect status and distribution source. No independent final validation is run: the supplied or stored final check is reused. See the quality vignette for interpretation.

Value

An object of class sspice_correction_quality_report.

Count the admissible data patterns induced by a routing DAG

Description

Uses dynamic programming and arbitrary-precision integers to count all record patterns compatible with the modal transitions represented by Bt.

Usage

count_admissible_patterns(
  dag,
  root_variables = NULL,
  max_memo_states = 5000000L,
  verbose = TRUE
)

Arguments

Details

Inspect n_admissible_patterns and n_all_unrestricted_patterns, stored as arbitrary-precision integers. Use as.character() to display their full values without conversion to floating point. n_memo_states describes the memoized computation. Counts concern structural patterns represented by the DAG, not all possible numeric or textual values in the original questionnaire.

Value

An object of class sspice_admissible_pattern_count.

Examples

routes <- data.frame(V1 = c("ID", "Q", "END"),
  V2 = c(1, 2, 1), V3 = c(1, 1, 1), V4 = c(9999, 2, 1),
  V5 = 0, V6 = 0, V7 = NA, V8 = NA)
dag <- build_DAG(routes, analyze = FALSE)
x <- data.frame(ID = 1:2, Q = c(1, 2), END = c(1, 1))
counts <- count_admissible_patterns(dag, verbose = FALSE)
as.character(counts$n_admissible_patterns)

Create the adjacency matrix from a transposed incidence matrix

Description

Builds a directed adjacency matrix using the signs of the incidence coefficients: +1 identifies a source node and -1 identifies a target node. This also preserves the direction of backward edges, allowing cycles to be detected correctly.

Usage

create_adiacency_matrix(B_transposed, vnames)

Arguments

Value

A square directed adjacency matrix.

Create logical-edit auxiliary variables

Description

Builds LEDIT variables from nine-column metadata. Each metadata row contains one base variable. Logical and arithmetic expressions are assembled across consecutive rows. LEDIT variables are appended to the input data.

Usage

create_logical_edit_variables(
  inpdata,
  logical_edits_data,
  extDFpath = NULL,
  separator = ";"
)

Arguments

Details

Every edit must define categories 1 and 2; both are accepted outcomes. Conditions are evaluated in file order and the first assignment wins. Unassigned records retain NA. Arithmetic missing operands are replaced by zero; missing logical outcomes do not match. These semantics differ from VAUX arithmetic. The metadata vignette provides a complete example. The returned data have a logical_edit_definitions attribute containing expressions, true-condition counts and newly assigned counts.

Value

The input data with the LEDIT variables appended.

Extend routing metadata with terminal logical-edit nodes

Description

Appends one routing node for each LEDIT defined in a nine-column metadata table. Every LEDIT has the accepted domain 1 and 2.

Usage

extend_routing_with_logical_edits(
  routing,
  logical_edits,
  separator = ";",
  na_strings = c("", ".", "NA", "NULL"),
  output_file = NULL
)

Arguments

Value

Extended routing data frame.

Find isolated graph nodes

Description

Identifies nodes having neither incoming nor outgoing edges.

Usage

find_isolated_nodes(A)

Arguments

Value

An integer vector containing the indices of isolated nodes.

Find a longest path in a DAG

Description

Finds a longest directed path between two selected nodes of a directed acyclic graph.

Usage

find_longest_path(A, vnames = NULL, start_node = 1L, end_node = nrow(A))

Arguments

Value

A list containing path, length, edge_length, and exists_path. For backward compatibility, length is the number of nodes in the path; edge_length is the number of traversed edges.

Generate records admissible under an SSPICE routing DAG

Description

Internal generator used to create controlled data for routing and localization tests.

Usage

generate_admissible_records(
  dag,
  n_records = 5000L,
  id_col = NULL,
  items_mapping_file = NULL,
  seed = 123L,
  optional_fill_probability = 0.5,
  max_attempts = 1000L
)

Arguments

Value

A data frame whose columns follow dag$varnames.

Inject controlled violations into admissible routing data

Description

Inject controlled violations into admissible routing data

Usage

inject_routing_violations(
  data,
  dag,
  id_col,
  items_mapping_file = NULL,
  proportion = 0.5,
  seed = 123L
)

Arguments

Value

A list with altered data and record-level ground truth.

Localize violations detected by SSPICE

Description

Internal engine that converts the complete output of routing_check() into localization information for validation and iterative repair. An existing routing-check result is reused without dropping logical-edit violations.

Usage

localize_routing_violations(
  data = NULL,
  dag = NULL,
  id_col = NULL,
  items_mapping_file = NULL,
  routing_results = NULL
)

Arguments

Value

An object of class sspice_violation_localization.

Prepare record-level data for SSPICE validation

Description

Creates auxiliary variables, coarsens logical OR and XOR groups, converts the resulting variables to the numeric representation used by SSPICE, and verifies their alignment with a graph created by build_DAG.

Usage

prepare_data(
  data,
  dag,
  vaux = NULL,
  external_data_dir = file.path("data-raw", "external"),
  data_separator = ",",
  metadata_separator = ";",
  na_strings = c("", ".", "NA", "NULL")
)

Arguments

Details

data_original retains the input; data_with_VAUX, data_with_VOR, and data_with_VOR_and_VXOR retain intermediate representations. vaux_metadata holds auxiliary rules and varnames gives the aligned names. Use result$data for checking. Text can become presence indicators and logical-group members can become canonical nodes; the final data are not a lossless copy of the questionnaire.

Value

An object of class sspice_prepared_data. Its data element is the final numeric data frame aligned with the DAG. Intermediate results are retained in the remaining elements.

Examples

routes <- data.frame(V1 = c("ID", "Q", "END"),
  V2 = c(1, 2, 1), V3 = c(1, 1, 1), V4 = c(9999, 2, 1),
  V5 = 0, V6 = 0, V7 = NA, V8 = NA)
dag <- build_DAG(routes, analyze = FALSE)
x <- data.frame(ID = 1:2, Q = c(1, 2), END = c(1, 1))
prepared <- prepare_data(x, dag)
prepared$data

Print an SSPICE DAG

Description

Print an SSPICE DAG

Usage

## S3 method for class 'sspice_dag'
print(x, ...)

Arguments

Value

x, invisibly.

Print prepared SSPICE data

Description

Print prepared SSPICE data

Usage

## S3 method for class 'sspice_prepared_data'
print(x, ...)

Arguments

Value

x, invisibly.

Check record conformity with an SSPICE routing structure

Description

Runs the path-based routing check and optionally adds an exhaustive scan of logical-edit variables. The latter reports every LEDIT equal to NA, even when an earlier path violation prevents that node from being reached.

Usage

routing_check(
  dset_data,
  routes,
  Bt,
  varkey,
  items.maps,
  path.items.maps,
  logical_edit_names = NULL
)

Arguments

Details

full_report has one row per reported violation, with the record key, variable labels, error code and message. summary_by_rec groups by identifier and message, so multiple rows can represent one record. summary_by_var groups by variable and message. Consistent records are absent from these summaries. violation_source is added to the full report only when logical-edit names are supplied. Item mapping translates dataset codes to structural routing codes without modifying the input records. See the routing guide for the mapping contract.

Value

A list containing full_report, summary_by_rec, summary_by_var, record_id, and logical_edit_names.

Examples

routes <- data.frame(V1 = c("ID", "Q", "END"),
  V2 = c(1, 2, 1), V3 = c(1, 1, 1), V4 = c(9999, 2, 1),
  V5 = 0, V6 = 0, V7 = NA, V8 = NA)
dag <- build_DAG(routes, analyze = FALSE)
x <- data.frame(ID = 1:2, Q = c(1, 2), END = c(1, 1))
mapping <- tempfile(fileext = ".csv")
write.csv(data.frame(varname = character(), item_route = numeric(),
                    item_dataset = numeric()), mapping, row.names = FALSE)
checked <- routing_check(x, dag$routes_numeric, dag$Bt, "ID",
                         basename(mapping), dirname(mapping))
checked$full_report
unlink(mapping)

Validate an extended canonical representation with ELOG variables

Description

Selects empirical semantic edits, constructs their ELOG variables with accepted domain 1 and 2, and verifies localization against the induced semantic ground truth.

Usage

validate_extended_elog(
  data,
  routing,
  id_col = "ID_RECORD",
  seed = 123L,
  min_outcome_count = 5L,
  max_search = 50000L,
  min_non_missing_rate = 0.25,
  max_levels = 12L
)

Arguments

Value

An object of class sspice_elog_validation.

Validate the Horn theory induced by a routing DAG

Description

Builds the elementary Horn basis induced by the modal rows of the transposed incidence matrix, verifies empirical closure equivalence, and evaluates rule indispensability through a deletion test.

Usage

validate_horn_theory(
  dag,
  rules_file = NULL,
  items_mapping_file = NULL,
  n_random = 5000L,
  max_seed_size = 8L,
  seed = 123L,
  add_bridges = TRUE
)

Arguments

Details

Inspect summary, canonical_tests, closure_tests, closure_failures, minimality_tests, and stored_rule_comparison. Rule tables are retained in elementary_rules, closure_basis and complete_rule_set. Random multi-seed closure checks provide empirical evidence, not an exhaustive proof over all subsets. The deletion tests assess rule indispensability.

Value

An object of class sspice_horn_validation.

Validate violation localization against ground truth

Description

Internal SSPICE function used to evaluate record-level localization when a validated ground-truth report is available. With report = NULL, it returns empty validation tables without interrupting the workflow.

Usage

validate_violation_localization(localization, report = NULL, id_col = NULL)

Arguments

Value

A list containing comparisons, metrics, and normalized truth values.