---
title: "Getting started with coreval"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with coreval}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

If you build SDTM datasets, you know the loop: write code, export, upload to a
validation tool, wait, read the report, work out which line of code caused each
finding, fix, repeat.

Hardly any of that time is spent *fixing* things. It goes on **finding out
what's broken**.

coreval does the finding part on your machine, in seconds. You still run your
qualified tool before you submit. You just arrive with a lot less for it to
find.

> coreval is a personal open-source project. It's not a CDISC product, isn't
> affiliated with or endorsed by CDISC, and isn't qualified or validated
> software. Treat every result as a hint, not a verdict. Your qualified tool
> and your own review are still what decide whether data is good to go.

```{r setup}
library(coreval)
```

## Start with one dataset

This is the one you'll use most while writing code. You have a data frame;
check it.

Note row 2: `2024-02-30`. February never has 30 days.

```{r one-dataset}
ae <- data.frame(
  STUDYID = "DEMO", DOMAIN = "AE", USUBJID = c("S1", "S1", "S2"),
  AESEQ = c(1, 2, 1), AETERM = c("Headache", "Nausea", "Rash"),
  AESTDTC = c("2024-01-10", "2024-02-30", "2024-01-12"),
  AEENDTC = c("2024-01-12", "2024-02-01", "")
)

result <- check_dataset(ae)
result
```

Each problem is described in words, with the rows and values that caused it, and
the rule number at the end in case you want to look it up.

The tag on each problem is worth understanding. CDISC Open Rules carry no
severity field - Pinnacle 21's Notes/Minor/Major/Critical is P21's own layer,
not CDISC's - so coreval does not report one and will not invent one. What it
does instead is separate the findings that are definitely wrong from the ones
that may be fine:

* `wrong value` - the data contains something that breaks the rule, like a
  month of 13. Nothing about your study explains it away. Start here.
* `missing required` - something the standard marks Required is absent.
* `missing optional` - something Expected is absent, or a value is blank. Often
  legitimate: a screen-failure subject with no reference dates, a variable your
  raw data does not carry yet.

Problems are ordered by that first and by how many records they touch second,
and within a problem the record holding a real offending value is shown before
one that is merely empty. It is also a `triage` column on every finding, so you
can sort a spreadsheet by it.

The rows themselves, ready to filter or count, are in
`result$findings`, with the same description in an `issue` column:

```{r one-dataset-date}
result$findings[result$findings$Value == "2024-02-30", ]
```

You can pass a file instead of a data frame: `.xpt`, `.sas7bdat` or `.csv`:

```{r one-dataset-file, eval = FALSE}
result <- check_dataset("ae.xpt")
```

coreval works out the domain from your `DOMAIN` column, and falls back to the
file name only when the data has no `DOMAIN` column at all. That order matters
for a split dataset: `ae1.xpt` is checked as `AE` because its `DOMAIN` column
says `AE`. On a file with no `DOMAIN` column the name `ae1` is taken at face
value. If it guesses wrong, just say so: `check_dataset(ae, domain = "AE")`.

### What it couldn't check, and why that matters

Lots of CDISC rules compare **one dataset against another**: an adverse event
date against the subject's reference dates in `DM`, a visit against the trial
design. Give coreval a single dataset and those questions simply can't be
answered.

coreval won't guess. It skips them and tells you what it wanted:

```{r one-dataset-skips}
cross <- result$skipped[grepl("was not supplied", result$skipped$reason), ]
nrow(cross)
head(unique(cross$reason), 3)
```

If it ran those anyway, it would be comparing your data against columns that
aren't there, and reporting problems that don't exist. Saying nothing is better
than making something up.

Nine rules ask a different kind of question: is this value one of the terms
CDISC's controlled terminology allows for this variable? `SEX` may be `F`, `M`,
`U` or `INTERSEX` and nothing else. Those need to know which version of the
terminology your study follows, because it changes between releases - `SEX`
gained `INTERSEX` and lost `UNDIFFERENTIATED`. So tell it:

```{r ct, eval = FALSE}
check_study(dir, ct_package = "sdtmct-2026-03-27")
list_ct_packages("sdtm")   # every published version, pick the one you declare
```

Without it those rules are reported as skipped, by name, saying exactly that -
coreval will not pick a version for you, because judging your data against
terminology it never declared would invent violations and hide real ones.

Most rules do still run. Across AE, DM, LB and VS, 76–84% of the applicable
ones work on a single dataset. But the ones that can't are the cross-dataset
checks, and those are often the ones you care about.

**So a short findings list here doesn't mean your data is clean.** It's a quick
first pass, not a verdict.

## Then check the whole study

Once the datasets exist as files, point coreval at the folder. Here's a small
one, built on the fly so this vignette runs without any data of your own:

```{r build-study}
dir <- tempfile("coreval_demo_")
dir.create(dir)

dm <- data.frame(
  STUDYID = "DEMO", DOMAIN = "DM", USUBJID = c("S1", "S2", "S3"),
  RFSTDTC = c("2024-01-05", "2024-01-06", ""),
  AGE = c(34, 61, 47), AGEU = c("YEARS", "YEARS", ""),
  SEX = c("M", "F", "F")
)

haven::write_xpt(dm, file.path(dir, "dm.xpt"))
haven::write_xpt(ae, file.path(dir, "ae.xpt"))
```

Point it at the **folder**, not a file:

```{r check}
study_result <- check_study(dir)
```

coreval reads everything in there, and reading it all at once is the point.
now the cross-dataset rules have both halves to work with. If there's a
Define-XML in the folder it finds it and uses it (that needs the `xml2`
package).

If you want to look at what was parsed, or check the same large study more than
once without re-reading it, do the read yourself and pass the object instead:

```{r read}
study <- read_study(dir)
names(study$datasets)
```

## Reading the results

You get two tables back, and you want to look at both.

### What's wrong

```{r findings-head}
head(study_result$findings)
```

One row per affected record, pointing straight at it:

| Column | What it tells you |
|---|---|
| `Dataset` | which dataset, or `STUDY` for whole-study checks |
| `Record` | row number, counting from 1 |
| `Variable` | the variable being complained about |
| `Value` | what was actually in there |
| `issue` | what's wrong, in words |
| `triage` | `wrong value`, `missing required` or `missing optional` |
| `rule_id` | the CDISC rule, if you want to look it up |

One thing that surprises people: `Not in dataset` under `Value` means the rule
wanted a variable you don't have, which is usually the finding.

It's a plain data frame, so slice it however you like:

```{r findings-filter}
f <- study_result$findings
head(f[f$Dataset == "DM", ])
sort(table(f$rule_id), decreasing = TRUE)[1:3]
```

Slicing the data frame gives you rows. `filter_findings()` gives you back a
*result*, the same object narrowed. That is the difference that matters:
because a result is what `write_findings()` and `summary()` take:

```{r filter-findings}
worst <- filter_findings(study_result, triage = "wrong value")
nrow(worst$findings)
summary(worst)
```

The two numbers there are counting different things, and the difference is
worth knowing: `findings` has a row per affected *record*, while `summary()`
counts distinct *problems*: one rule going wrong in one dataset, however many
records it touched. A single missing variable in a 500-row dataset is one
problem and 500 rows.

`triage` has three levels, worst first: `"wrong value"` is a value that
contradicts the data around it, `"missing required"` a variable the standard
says must be there, `"missing optional"` one it merely expects. You can also
narrow by `dataset`, `rule` or `variable`, and combine them.

### What couldn't be checked

```{r skipped}
head(study_result$skipped)
```

**This is the table people skip, and it's the one that bites.** An empty
findings table means one of two things: your data is clean, or a lot of rules
never ran. Those look identical if you only read the findings. coreval always
shows you both, with a reason for every rule it couldn't run.

### Saving it, and tracking what you didn't fix

```{r export, eval = FALSE}
write_findings(study_result, "issues.xlsx")  # one workbook, a sheet per table
write_findings(study_result, "issues.csv")   # issues.csv + _skipped + _about
```

Both tables get written every time, for the reason just above. Excel output
needs the `writexl` package.

The saved file has three empty columns, `Status`, `Owner` and `Notes`, for you
to fill in once it is open. Not every finding is something you will fix: some are
expected, some belong to someone else, some are waiting on a data query. Those
decisions are worth recording next to the finding rather than in a separate
document. Pass `tracking = FALSE` if you would rather not have them.

## The rule set

One function answers every question about the rule set.

```{r rules}
rules <- list_rules()
nrow(rules)
table(rules$source)

attr(rules, "rules_version")
```

That last one is the exact CDISC commit the bundled rules came from. Worth
writing down next to your results, though `write_findings()` already records
it in every file it saves.

Not every rule carries the same weight. `source` separates fully-vetted
published rules from deprecated and draft ones; `?list_rules` says what each
means.

Note the count above includes the deprecated ones. `list_rules()` is the
catalog of what is *bundled*; a check excludes superseded rules, because
running one alongside its replacement reports the same problem twice. Listing
is not running.

Ask it what applies to a domain, or what a rule the report named actually
checks:

```{r rules-for-domain}
nrow(list_rules(domain = "AE"))

rule <- list_rules(id = "CORE-000547")
rule$issue
```

Two more columns are worth knowing. `legacy_ids` are the names Pinnacle 21 and
CDISC's older Conformance Rules spreadsheets use for the same rule, which is
how you match a finding here to a line in a P21 report. `guidance` is the
sentence from the Implementation Guide the rule exists to enforce: the reason
behind it.

```{r rule-ids}
rule$legacy_ids
```

`print(result, guidance = TRUE)` shows that sentence under each problem in the
report. It's off by default because it roughly doubles the length.

## Narrowing to your standard

With no standard given, coreval runs the rules for every standard it has, and
the report says so. Tell it which one your data follows and it runs only
those, and only for your version of the Implementation Guide:

```{r narrow, eval = FALSE}
result <- check_dataset(dm, standard = "SDTMIG", version = "3.4")
```

That cuts the list a lot, and it can cut too far. CDISC's coverage is uneven:
the general "dates must be valid ISO 8601" rule is published for SEND and the
Tobacco Implementation Guide but not for SDTM, so narrowing to SDTMIG stops a
month of 13 being reported. The report always says how many rules it set
aside. Leave `standard` unset if you'd rather see everything.

## USDM study designs

A USDM study file is a single JSON document describing a study design rather
than a folder of datasets. Put it in a folder and check the folder, the same
way as any other study. It needs the `jsonlite` and `QuickJSR` packages.

```{r usdm, eval = FALSE}
result <- check_study("path/to/usdm")
```

Problems are reported against the part of the design they're about, such as
`STUDYDESIGN` or `ENCOUNTER`, and the row within it.

## The whole API

Seven functions, and three of them do the work:

| | |
|---|---|
| `check_dataset(x)` | one dataset: a data frame, or an `.xpt`/`.sas7bdat`/`.csv` |
| `check_study(path)` | a whole folder |
| `write_findings(result, path)` | save to Excel or CSV |
| `list_rules()` | the rule set, one rule, or the rules for a domain |
| `filter_findings(result, ...)` | narrow a result |
| `read_study(path)` | read a folder yourself, when you want to inspect it |
| `list_ct_packages()` | the Controlled Terminology releases you can pass as `ct_package` |

Plus `print()` and `summary()` on a result. `print()` is what you get by
typing the result's name; `summary()` you call yourself, and it returns a
one-row table you can rbind across datasets.

## Two last things

**Nothing leaves your machine.** No internet, no API key, no account. The rules
and the standards metadata are bundled inside the package.

**Think of the accuracy number as a floor.** For most rules CDISC publishes an
answer sheet: some example data, and the exact rows a correct implementation
should flag. Every rule here is run against those examples and compared row by
row, and more than nine in ten come back with exactly the answer CDISC gives.

But those examples are small and tidy, and real submissions are neither, so
agreement is a lower bound on correctness rather than a score. The README lists
what coreval doesn't do yet. The advice does not change: run your
qualified tool before you submit.

```{r cleanup, include = FALSE}
unlink(dir, recursive = TRUE)
```