Most link-graph analysis stops at the diagnosis. This vignette runs the whole loop on one small site: a problem was found in a crawl, a fix was modeled, the fix shipped, and the site was crawled again. Both crawls ship with the package, so every number below is computed as you read it rather than quoted from a report.
The site is a 67-page reviews microsite. The diagnosis was that four
sitewide in-content links — methodology and disclosure bylines repeated
under every article — were hoarding editorial authority and returning
none of it. The fix had two parts: move those bylines into a semantic
<nav> so they stop counting as content, and add
genuine body links into the most-orphaned commercial pages.
The interesting result is not that it worked. It is that whether you can see it working at all depends entirely on which graph you score.
The fixture is two Screaming Frog exports per phase —
internal_all.csv for the page inventory and
all_inlinks.csv for the links.
crawl <- function(phase) {
dir <- system.file(
"extdata", paste0("reviews-microsite-", phase),
package = "pagerankr"
)
screaming_frog_bundle(
internal = file.path(dir, "internal_all.csv"),
links = file.path(dir, "all_inlinks.csv"),
link_export_kind = "all_inlinks"
)
}
before <- crawl("before")
after <- crawl("after")
c(before = nrow(before$edges), after = nrow(after$edges))
#> before after
#> 3820 3599The site is pseudonymous: hosts, paths, anchors and titles were
replaced by a deterministic relabeling, while everything structural —
status codes, crawl depth, link counts, Link Position and
Link Path — is verbatim. Paths like
/s2/p06/p05/ preserve depth and sibling grouping and carry
no meaning. inst/extdata/README.md documents the scheme in
full.
pagerank_screaming_frog() scores the whole graph by
default. Passing accepted_placements = "Content" restricts
it to links Screaming Frog placed in the body — the
editorial graph, the subset a human actually chose per
page.
score <- function(bundle, ...) pagerank_screaming_frog(bundle, ...)
ed_before <- score(before, accepted_placements = "Content")
ed_after <- score(after, accepted_placements = "Content")
full_before <- score(before)
full_after <- score(after)External hosts get scored too. For site-level reporting we want internal pages only, but note that rank positions below are taken over the full scored set, externals included — that is the ranking the pipeline actually produces, and trimming it first would quietly renumber everything.
internal <- function(x) {
x[grepl("reviews-microsite", x$node_name, fixed = TRUE), ]
}
c(
editorial_before = nrow(internal(ed_before)),
editorial_after = nrow(internal(ed_after)),
full_before = nrow(internal(full_before)),
full_after = nrow(internal(full_after))
)
#> editorial_before editorial_after full_before full_after
#> 67 62 67 62Sixty-seven pages before, sixty-two after. Five pages were retired during the work. Hold that number — it is the main reason the naive comparison is a trap.
Worth stating before any share is computed, because it silently breaks the obvious arithmetic:
c(
editorial_before = sum(internal(ed_before)$pagerank),
editorial_after = sum(internal(ed_after)$pagerank)
)
#> editorial_before editorial_after
#> 0.8235217 0.8885489Mass that reaches a page with no onward links leaves the distribution
rather than being recycled; pagerankr accounts for it
separately as wasted mass instead of quietly redistributing it. So
every share must be taken against
sum(pagerank), never assumed to be 1.
That the total rose is itself part of the result: draining the sinks meant less mass ended up stranded.
A de-sink makes one specific prediction — authority stops piling onto a few hoarding nodes and spreads across the rest. Concentration metrics test exactly that, and they are robust to the node set changing underneath them.
top_n_share <- function(x, n = 5) {
sum(sort(x, decreasing = TRUE)[seq_len(n)]) / sum(x)
}
concentration <- function(x) {
pr <- internal(x)$pagerank
c(
n = length(pr),
gini = round(pr_gini(pr), 3),
entropy = round(pr_entropy(pr), 2),
top5_share = round(100 * top_n_share(pr), 1)
)
}
rbind(
before = concentration(ed_before),
after = concentration(ed_after)
)
#> n gini entropy top5_share
#> before 67 0.715 2.70 71.0
#> after 62 0.589 3.39 47.7Gini fell from 0.715 to 0.589, entropy rose, and the top five pages went from holding about 71% of editorial authority to about 48%.
Note top_n_share() is written by hand above.
pr_top_k_share() exists, but its k is a
fraction of nodes, not a count —
pr_top_k_share(x, 0.1) is the top 10%. Passing
5 is an error, not a top-5.
The largest single lever was not the links that were added. It was the links that stopped counting.
Moving the bylines into
<nav aria-label="Editorial standards"> makes
Screaming Frog record their Link Position as
Navigation instead of Content. They do not
lose weight — they leave the editorial graph
entirely.
content_edges <- function(bundle) {
e <- bundle$edges
e <- e[!is.na(e$link_position) & e$link_position == "Content", ]
keep <- grepl("reviews-microsite", e$from, fixed = TRUE) &
grepl("reviews-microsite", e$to, fixed = TRUE)
nrow(unique(e[keep, c("from", "to")]))
}
c(before = content_edges(before), after = content_edges(after))
#> before after
#> 406 307Ninety-nine internal content edges disappeared, and almost none of
the HTML changed. This is the practical corollary of the boilerplate
problem — see vignette("boilerplate") — in-content template
links are a second navigation, and the cheapest correct fix is to make
the markup say so.
Now score both crawls without the placement filter.
full <- merge(
ranked(full_before), ranked(full_after),
by = "url", suffixes = c("_before", "_after")
)
full <- full[grepl("reviews-microsite", full$url, fixed = TRUE), ]
c(
pages = nrow(full),
gini_before = round(pr_gini(internal(full_before)$pagerank), 3),
gini_after = round(pr_gini(internal(full_after)$pagerank), 3),
pearson = round(cor(full$pr_before, full$pr_after), 4)
)
#> pages gini_before gini_after pearson
#> 62.0000 0.2970 0.2570 0.9999Pearson correlation of 0.9999 across the pages present in both crawls. To four decimal places, the full-graph ranking is the same vector before and after.
An intervention that moved a page from #1 to #38 on one lens is invisible on the other. The site has roughly 3,300 navigation edges against 400 content ones, so the template dominates the full graph and swamps any editorial change. The lens decides whether the change exists at all — which is the argument for running the editorial view as a matter of course, not as a special case.
This is a two-crawl natural experiment, not a controlled simulation on a fixed graph. Three limits apply, and stating them is part of the method:
The node set changed. Five pages were retired between crawls. Absolute editorial PageRank is therefore not comparable across the pair — read rank shifts, concentration metrics, and relative deltas instead.
Full-graph absolute levels drift for a mechanical reason. Individual pages rise 1–23% in the full graph purely because those five pages left and their mass redistributed. That spread is a node-set artifact, not a response to the intervention. The correlation is the honest statistic; the per-page percentages there are not.
Two changes shipped together. The byline
reclassification and the new feeder links landed in the same release, so
what is measured is the net effect. Separating them requires modeling
each on a fixed graph — which is what simulate_changes() is
for.
On that last point: the pre-ship simulate_changes()
projection agreed with this re-crawl in direction and rough magnitude.
That agreement, on a real site, is the strongest claim available — the
diagnosis was not merely internally consistent, it predicted something
that then happened.
The whole method is four decisions:
Internal > All and
Bulk Export > Links > All Inlinks at both time
points, and keep Link Position and
Link Path.accepted_placements = "Content", once without. Differences
between the two lenses are the finding, not noise.sum(pagerank) rather than assuming it is 1, and
match pages on the full URL.For the modeling half of the loop — projecting a change before
shipping it — see vignette("pagerankr-usage") and
simulate_changes(). For the placement weighting used to
soften rather than exclude template links, see
vignette("presets").