Package {RKorAPClient}

R Client for KorAP Corpus Analysis Platform

Description

RKorAPClient provides programmatic access to KorAP corpus analysis platform instances, enabling corpus linguistic research on large corpora like DeReKo, CoRoLa, DeLiKo@DNB.

Main Functions

Connection

KorAPConnection(), auth(), persistAccessToken()

Search

corpusQuery(), fetchAll(), fetchNext()

Analysis

corpusStats(), frequencyQuery(), collocationAnalysis()

Quick Start

library(RKorAPClient)
# Connect and search
kcon <- KorAPConnection()
query <- corpusQuery(kcon, "Ameisenplage")
results <- fetchAll(query)

# Access results
results@collectedMatches
results@totalResults

Common Workflows

Basic Search:

kcon <- KorAPConnection()
kcon |> corpusQuery("search term") |> fetchAll()

Frequency Analysis:

frequencyQuery(kcon, c("term1", "term2"), vc="pubDate in 2020")

Corpus Statistics:

corpusStats(kcon, vc="textType=Zeitung", as.df=TRUE)

Author(s)

Maintainer: Marc Kupietz kupietz@ids-mannheim.de

Authors:

• Marc Kupietz kupietz@ids-mannheim.de

Other contributors:

• Nils Diewald diewald@ids-mannheim.de [contributor]

• Leibniz Institute for the German Language [copyright holder, funder]

References

Kupietz, Marc / Diewald, Nils / Margaretha, Eliza (2020): RKorAPClient: An R package for accessing the German Reference Corpus DeReKo via KorAP. In: Calzolari, Nicoletta, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis (eds.): Proceedings of The 12th Language Resources and Evaluation Conference (LREC 2020) Marseille: European Language Resources Association (ELRA), 7017-7023. http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.867.pdf

See Also

Useful links:

• https://github.com/KorAP/RKorAPClient/

• https://korap.ids-mannheim.de/

• https://www.ids-mannheim.de/digspra/kl/projekte/korap

• Report bugs at https://github.com/KorAP/RKorAPClient/issues

Connect to KorAP Server

Description

KorAPConnection() creates a connection to a KorAP server for corpus queries. This is your starting point for all corpus analysis tasks.

Usage

KorAPConnection(
  KorAPUrl = defaultKorAPUrl(),
  apiVersion = "v1.0",
  apiUrl,
  accessToken = getAccessToken(KorAPUrl),
  oauthClient = NULL,
  oauthScope = "search match_info",
  authorizationSupported = TRUE,
  userAgent = "R-KorAP-Client",
  timeout = 240,
  verbose = FALSE,
  cache = TRUE
)

Arguments

kco <- KorAPConnection(accessToken="<access token>")

persistAccessToken(kco)

Details

Use KorAPConnection() to connect, then corpusQuery() to search, and fetchAll() to retrieve results. For authorized access to restricted corpora, use auth() or provide an accessToken.

The KorAPConnection object contains various configuration slots for advanced users: KorAPUrl (server URL), apiVersion, accessToken (OAuth2 token), timeout (request timeout), verbose (logging), cache (local caching), and other technical parameters. Most users can ignore these implementation details.

Value

KorAPConnection() object that can be used e.g. with corpusQuery()

Basic Workflow

# Connect to KorAP
kcon <- KorAPConnection()

# Search for a term
query <- corpusQuery(kcon, "Ameisenplage")

# Get all results
results <- fetchAll(query)

Authorization

For access to restricted corpora, authorize your connection:

kcon <- KorAPConnection() |> auth()

See Also

Other initialization functions: auth,KorAPConnection-method, clearAccessToken,KorAPConnection-method, persistAccessToken,KorAPConnection-method

KorAPCorpusStats class (internal)

Description

Internal class for corpus statistics storage. Users work with corpusStats() function instead.

Usage

## S4 method for signature 'KorAPCorpusStats'
show(object)

Arguments

KorAPQuery class (internal)

Description

Internal class for query state management. Users work with corpusQuery(), fetchAll(), and fetchNext() instead.

Usage

buildWebUIRequestUrlFromString(KorAPUrl, query, vc = "", ql = "poliqarp")

buildWebUIRequestUrl(
  kco,
  query = if (missing(KorAPUrl)) {
    
    stop("At least one of the parameters query and KorAPUrl must be specified.", call. =
    FALSE)
 } else {
     httr2::url_parse(KorAPUrl)$query$q
 },
  vc = if (missing(KorAPUrl)) "" else httr2::url_parse(KorAPUrl)$query$cq,
  KorAPUrl,
  ql = if (missing(KorAPUrl)) "poliqarp" else httr2::url_parse(KorAPUrl)$query$ql
)

## S3 method for class 'KorAPQuery'
format(x, ...)

## S4 method for signature 'KorAPQuery'
show(object)

Arguments

Internal API call method

Description

Internal API call method

Usage

## S4 method for signature 'KorAPConnection'
apiCall(
  kco,
  url,
  json = TRUE,
  getHeaders = FALSE,
  cache = kco@cache,
  timeout = kco@timeout,
  verbose = kco@verbose
)

Arguments

Association score functions

Description

Functions to calculate different collocation association scores between a node (target word) and words in a window around the it. The functions are primarily used by collocationScoreQuery().

pmi: pointwise mutual information

mi2: pointwise mutual information squared (Daille 1994), also referred to as mutual dependency (Thanopoulos et al. 2002)

mi3: pointwise mutual information cubed (Daille 1994), also referred to as log-frequency biased mutual dependency) (Thanopoulos et al. 2002)

logDice: log-Dice coefficient, a heuristic measure that is popular in lexicography (Rychlý 2008)

ll: log-likelihood (Dunning 1993) using Stefan Evert's (2004) simplified implementation

Usage

defaultAssociationScoreFunctions()

pmi(O1, O2, O, N, E, window_size)

mi2(O1, O2, O, N, E, window_size)

mi3(O1, O2, O, N, E, window_size)

logDice(O1, O2, O, N, E, window_size)

ll(O1, O2, O, N, E, window_size)

Arguments

Details

logDice is computed as defined by Rychlý (2008), that is from the plain marginal frequencies of node and collocate, so that its values are comparable to those of other tools, such as Sketch Engine. Unlike the expectation based scores, it does not take the window size into account: the Dice coefficient relates the co-occurrence frequency to how often the two words occur at all, and O1 and O2 count word tokens, while a window size factor would count window positions.

Value

         association score

Salience versus surprise

logDice is the only one of these scores that does not compare the observed co-occurrence frequency to an expected one. As it merely puts O into the numerator of the Dice coefficient, the difference between the score of a pair and the score that same pair would get if it co-occurred exactly as often as expected is precisely the pointwise mutual information:

logDice(O) - logDice(E) = log2(O / E) = pmi

The level a pair starts from therefore depends on the marginal frequencies alone, and it is high whenever both words are frequent. Collocates of Grund in DeReKo, in a window of five words to each side, illustrate this:

• triftiger is rare, so it starts from a logDice of -4.60 and reaches 3.96, exceeding what is expected by a pmi of 8.57

• Berlin is frequent, so it starts from 5.58 and still reaches 3.84, while co-occurring 1.74 bits less often than expected

A frequent collocate can thus reach a respectable logDice although the node does not attract it at all. logDice measures how salient a pair is, given how often its words occur, rather than how surprising it is. That is what it was designed for (Rychlý 2008), and it is why its values do not depend on the corpus size and are comparable across corpora.

Since collocationAnalysis() ranks and thresholds by logDice, it therefore drops collocates occurring less often than expected by default. Its minObservedExpectedRatio parameter controls this: raise it to demand a stronger contrast, or set it to 0 to see the unfiltered ranking, for instance to study repulsion. collocationScoreQuery() does not filter, as there the pairs to score are given explicitly. For results obtained otherwise, dplyr::filter(O > E) or a minimum pmi or ll has the same effect.

References

Daille, B. (1994): Approche mixte pour l’extraction automatique de terminologie: statistiques lexicales et filtres linguistiques. PhD thesis, Université Paris 7.

Thanopoulos, A., Fakotakis, N., Kokkinakis, G. (2002): Comparative evaluation of collocation extraction metrics. In: Proc. of LREC 2002: 620–625.

Rychlý, Pavel (2008): A lexicographer-friendly association score. In Proceedings of Recent Advances in Slavonic Natural Language Processing, RASLAN, 6–9. https://www.fi.muni.cz/usr/sojka/download/raslan2008/13.pdf.

Dunning, T. (1993): Accurate methods for the statistics of surprise and coincidence. Comput. Linguist. 19, 1 (March 1993), 61-74.

Evert, Stefan (2004): The Statistics of Word Cooccurrences: Word Pairs and Collocations. PhD dissertation, IMS, University of Stuttgart. Published in 2005, URN urn:nbn:de:bsz:93-opus-23714. Free PDF available from https://purl.org/stefan.evert/PUB/Evert2004phd.pdf

See Also

Other collocation analysis functions: collocationAnalysis,KorAPConnection-method, collocationScoreQuery,KorAPConnection-method, synsemanticStopwords()

Examples

## Not run: 

KorAPConnection(verbose = TRUE) %>%
collocationScoreQuery("Perlen", c("verziertes", "Säue"),
  scoreFunctions = append(defaultAssociationScoreFunctions(),
     list(localMI = function(O1, O2, O, N, E, window_size) {
                       O * log2(O/E)
                    })))

## End(Not run)

Authorize RKorAPClient

Description

Authorize RKorAPClient to make KorAP queries and download results on behalf of the user.

Usage

## S4 method for signature 'KorAPConnection'
auth(
  kco,
  app_id = generic_kor_app_id,
  app_secret = NULL,
  scope = kco@oauthScope
)

Arguments

Value

KorAPConnection object with access token set in ⁠@accessToken⁠.

See Also

persistAccessToken(), clearAccessToken()

Other initialization functions: KorAPConnection-class, clearAccessToken,KorAPConnection-method, persistAccessToken,KorAPConnection-method

Examples

## Not run: 
kco <- KorAPConnection(verbose = TRUE) %>% auth()
df <- collocationAnalysis(kco, "focus([marmot/p=ADJA] {Ameisenplage})",
  leftContextSize = 1, rightContextSize = 0
)

## End(Not run)

Vouch for a cacheAs file that an older version wrote

Description

Association scores changed in 1.4.0, so files from before it are recomputed rather than used (see cacheAs). Where a file is known to hold what this version would compute - because it was written by a development version that already had the corrections, for instance - this records that, and the file is used again as it is.

Usage

blessCacheAs(cacheAs)

Arguments

Details

What actually wrote a file is left as it stands; the blessing is recorded beside it, so that cacheAsInfo() keeps telling the truth about where the numbers come from.

A file that records no parameters, as those written before 1.3.0.9000 do, has nothing left to be compared against a call once it is blessed, and is therefore reused for any call that names it. Bless such a file only if that is what you mean.

Value

the paths, invisibly

See Also

Other cacheAs: cacheAsInfo(), withCachedResults()

Examples

## Not run: 
blessCacheAs("klima-ca.rds")
blessCacheAs(list.files("data", pattern = "\\.rds$", full.names = TRUE))

## End(Not run)

Keeping results in a file of one's own

Description

The cacheAs parameter of the query functions is a different thing from the cache parameter of KorAPConnection(). The latter is a transparent speed-up: it stores server responses where the package finds them again, and throwing it away costs nothing but time. The former stores a finished result in a file the caller names and keeps, which is what makes an analysis reproducible: KorAP corpora grow, so the same query returns different numbers next year, and the scores computed from them may change with the package.

Details

A cache file therefore records what produced it, and is not reused when that no longer matches what is being asked for.

What produced a cacheAs file

Description

Reads back what a query function recorded in a cacheAs file: the parameters it was called with, the KorAP instance it asked, the index revision that instance's corpus had at the time, and the version of RKorAPClient that wrote the file. Useful for saying, of a result kept next to a document, what the numbers in it rest on.

Usage

cacheAsInfo(cacheAs)

Arguments

Value

a list with the elements scoreVersion, packageVersion, parameters, dots, ignoredParameters (inactive options, when recorded), apiUrl and indexRevision, or NULL for a file written by a version before 1.4.0, which recorded none of this

See Also

Other cacheAs: blessCacheAs(), withCachedResults()

Examples

## Not run: 
KorAPConnection() |> frequencyQuery("Ameisenplage", cacheAs = "ameisenplage.rds")
cacheAsInfo("ameisenplage.rds")

## End(Not run)

Calculate and format ETA for batch operations

Description

Helper function to calculate estimated time of arrival based on elapsed time and progress through a batch operation.

Usage

calculate_eta(current_item, total_items, start_time)

Arguments

Value

character string with formatted ETA and completion time or empty string if not calculable

Add confidence interval and relative frequency variables

Description

Using prop.test(), ci adds three columns to a data frame:

1. relative frequency (f)

2. lower bound of a confidence interval (ci.low)

3. upper bound of a confidence interval

Convenience function for converting frequency tables to instances per million.

Convenience function for converting frequency tables of alternative variants (generated with as.alternatives=TRUE) to percent.

Converts a vector of query or vc strings to typically appropriate legend labels by clipping off prefixes and suffixes that are common to all query strings.

Experimental convenience function for plotting typical frequency by year graphs with confidence intervals using ggplot2. Warning: This function may be moved to a new package.

Usage

ci(df, x = totalResults, N = total, conf.level = 0.95)

ipm(df)

percent(df)

queryStringToLabel(data, pubDateOnly = FALSE, excludePubDate = FALSE)

geom_freq_by_year_ci(
  mapping = aes(ymin = conf.low, ymax = conf.high),
  showCI = TRUE,
  smooth = FALSE,
  ...
)

Arguments

Details

Given a table with columns f, conf.low, and conf.high, ipm ads a ⁠column ipm⁠ und multiplies conf.low and conf.high with 10^6.

With smooth = TRUE, map numeric calendar years to x and frequency to y. Interpolation is only a display aid, not a fitted trend or a trend test. Confidence-band boundaries are interpolated between the supplied annual limits; they are not confidence intervals for unobserved dates. Observed annual values and limits are retained. As with other ggplot2 statistics, interpolation follows the axis scales, including logarithmic scales. Cubic curves can overshoot between annual values; use straight connections when the exact shape matters. Negative interpolated frequencies are clipped at zero (on a log axis, zero is outside the displayed range). Each group needs at most one observation per year; use colour, fill or group to distinguish expressions or corpora. No extrapolation is performed. The interpolation used by Highcharts may give a different curve between the same annual observations.

Value

original table with additional column ipm and converted columns conf.low and conf.high

original table with converted columns f, conf.low and conf.high

string or vector of strings with clipped off common prefixes and suffixes

See Also

ci is already included in frequencyQuery()

Examples

## Not run: 

library(ggplot2)
kco <- KorAPConnection(verbose=TRUE)
expand_grid(year=2015:2018, alternatives=c("Hate Speech", "Hatespeech")) %>%
  bind_cols(corpusQuery(kco, .$alternatives, sprintf("pubDate in %d", .$year))) %>%
  mutate(total=corpusStats(kco, vc=vc)$tokens) %>%
  ci() %>%
  ggplot(aes(x=year, y=f, fill=query, color=query, ymin=conf.low, ymax=conf.high)) +
    geom_point() + geom_line() + geom_ribbon(alpha=.3)

## End(Not run)
## Not run: 

KorAPConnection() %>% frequencyQuery("Test", paste0("pubDate in ", 2000:2002)) %>% ipm()

## End(Not run)
## Not run: 

KorAPConnection() %>%
    frequencyQuery(c("Tollpatsch", "Tolpatsch"),
    vc=paste0("pubDate in ", 2000:2002),
    as.alternatives = TRUE) %>%
  percent()

## End(Not run)
queryStringToLabel(paste("textType = /Zeit.*/ & pubDate in", c(2010:2019)))
queryStringToLabel(c("[marmot/m=mood:subj]", "[marmot/m=mood:ind]"))
queryStringToLabel(c("wegen dem [tt/p=NN]", "wegen des [tt/p=NN]"))

## Not run: 
library(ggplot2)
kco <- KorAPConnection(verbose=TRUE)

expand_grid(condition = c("idsColumn = /Wirtschaft.*/", "idsColumn != /Wirtschaft.*/"),
            year = (2005:2011)) %>%
  cbind(frequencyQuery(kco, "[tt/l=Heuschrecke]",
                            paste0(.$condition," & pubDate in ", .$year)))  %>%
  ipm() %>%
  ggplot(aes(year, ipm, fill = condition, color = condition)) +
  geom_freq_by_year_ci()

## End(Not run)

Clear access token from keyring and KorAPConnection object

Description

Clear access token from keyring and KorAPConnection object

Usage

## S4 method for signature 'KorAPConnection'
clearAccessToken(kco)

Arguments

Value

KorAPConnection object with access token set to NULL.

See Also

persistAccessToken()

Other initialization functions: KorAPConnection-class, auth,KorAPConnection-method, persistAccessToken,KorAPConnection-method

Examples

## Not run: 
kco <- KorAPConnection()
kco <- clearAccessToken(kco)

## End(Not run)

Clear local cache

Description

Clears the local cache of API responses for the current RKorAPClient version. Useful when you want to force fresh data retrieval or free up disk space.

Usage

## S4 method for signature 'KorAPConnection'
clearCache(kco)

Arguments

Value

Invisible NULL (function called for side effects)

Examples

## Not run: 
kco <- KorAPConnection()
clearCache(kco)

## End(Not run)

Collocation analysis

Description

Performs a collocation analysis for the given node (or query) in the given virtual corpus.

Usage

## S4 method for signature 'KorAPConnection'
collocationAnalysis(
  kco,
  node,
  vc = "",
  lemmatizeNodeQuery = FALSE,
  minOccur = 5,
  leftContextSize = 5,
  rightContextSize = 5,
  topCollocatesLimit = 200,
  searchHitsSampleLimit = 20000,
  ignoreCollocateCase = FALSE,
  withinSpan = ifelse(exactFrequencies, "base/s=s", ""),
  exactFrequencies = TRUE,
  stopwords = append(RKorAPClient::synsemanticStopwords(), node),
  seed = 7,
  expand = length(vc) != length(node),
  maxRecurse = 0,
  addExamples = FALSE,
  thresholdScore = "logDice",
  threshold = 2,
  localStopwords = c(),
  collocateFilterRegex = "^[:alnum:]+-?[:alnum:]*$",
  minObservedExpectedRatio = 1,
  queryMissingScores = FALSE,
  missingScoreQuantile = 0.05,
  vcLabel = NA_character_,
  cacheAs = NULL,
  ...
)

Arguments

Details

The collocation analysis is currently implemented on the client side, as some of the functionality is not yet provided by the KorAP backend. Mainly for this reason it is very slow (several minutes, up to hours), but on the other hand very flexible. You can, for example, perform the analysis in arbitrary virtual corpora, use complex node queries, and look for expression-internal collocates using the focus function (see examples and demo).

To increase speed at the cost of accuracy and possible false negatives, you can decrease searchHitsSampleLimit and/or topCollocatesLimit and/or set exactFrequencies to FALSE.

Note that some outdated non-DeReKo back-ends might not yet support returning tokenized matches (warning issued). In this case, the client library will fall back to client-side tokenization which might be slightly less accurate. This might lead to false negatives and to frequencies that differ from corresponding ones acquired via the web user interface.

Value

A tibble where each row represents a candidate collocate for the requested node. Columns include (depending on the selected association measures):

• node, collocate, vc, label: identifiers for the query node, collocate, virtual corpus, and optional label.

• Frequency and contingency information such as frequency, O, O1, O2, E, leftContextSize, rightContextSize, and w.

• Association measures (e.g. logDice, ll, mi, ...), one column per requested scorer.

• Per-labelled association scores produced by multi-VC comparisons using the pattern <measure>_<label>.

• Ranks per label/measure with the pattern rank_<label>_<measure> (1 is best) and the corresponding percentile ranks percentile_rank_<label>_<measure>.

• Pairwise contrasts for two-label comparisons, e.g. delta_<measure>, delta_rank_<measure>, and delta_percentile_rank_<measure>.

• Summary columns describing the strongest labels per measure (winner_*, runner_up_*, loser_*, and max_delta_*), including winner/loser webUIRequestUrl columns. In multi-VC comparisons, missing per-label concordance URLs are derived from another available row URL for the same node/collocate by replacing the cq parameter with the target label's virtual corpus. Unsuffixed winner_webUIRequestUrl and loser_webUIRequestUrl columns are populated only when the score-based URL choices agree.

• imputed_<label>: whether the score for that label was imputed rather than observed (see missingScoreQuantile). n_imputed counts them, and imputed is n_imputed > 0: it describes the node/collocate pair across all labels, not the label of the row it stands in. A row can therefore carry imputed = TRUE while its own scores are perfectly attested, because the pair is missing from some other virtual corpus - use imputed_<label> for the row itself. Filter with dplyr::filter(!imputed) to keep only collocates attested in every compared virtual corpus.

• Optional helper columns such as query, example, or url when example retrieval is requested.

Interpreting multi-VC comparisons

The comparison columns produced when vc holds more than one virtual corpus are experimental: their names and semantics may still change in a future release without a deprecation cycle. Code that has to keep working across versions should select the columns it needs explicitly.

They are an exploration aid, not a significance test. When reading them, keep three properties in mind.

Imputed scores describe presence/absence, not contrast. A collocate that passes the minOccur and topCollocatesLimit thresholds in one virtual corpus but not in another has no observed score for the latter. Such cells are imputed from a floor derived from the pooled result set (see missingScoreQuantile), so the corresponding ⁠delta_*⁠ and ⁠max_delta_*⁠ values measure the distance to that floor rather than an attested difference. The imputed, n_imputed and ⁠imputed_<label>⁠ columns mark these rows; dplyr::filter(!imputed) restricts the result to collocates attested everywhere, and queryMissingScores = TRUE replaces most imputed cells with scores actually retrieved from the backend. Mind what imputed is about: the pair, not the row. It is TRUE as soon as one label lacks the collocate, and stays TRUE on the rows of the labels where it is attested, which is what makes dplyr::filter(!imputed) drop the pair as a whole. Whether the row at hand rests on an imputed score is what ⁠imputed_<label>⁠ says.

Per-label columns carry syntactic names. The label in ⁠<measure>_<label>⁠, ⁠rank_<label>_<measure>⁠ and ⁠imputed_<label>⁠ is the one the caller gave, put through make.names(), so that the result stays a well formed data frame: a virtual corpus named 1976-1980 appears as logDice_X1976.1980. The label column and the ⁠winner_*⁠ / ⁠loser_*⁠ columns keep the name as it was given, so mapping between the two means applying the same transformation, e.g. stats::setNames(make.names(labels), labels).

Imputed values are relative to one analysis. The floor is computed from the scores present in the result at hand. Analysing a node on its own and analysing it together with other nodes therefore yield different imputed values, and deltas involving imputed cells are not comparable across separate calls. Deltas between observed scores are unaffected.

Winners carry no uncertainty. Unlike ci(), which attaches confidence intervals to relative frequencies, the ⁠winner_*⁠ / ⁠loser_*⁠ columns simply order point estimates. A collocate wins by a hair on six occurrences exactly as decisively as one that wins by a wide margin on thousands. Consult the observed frequencies (O, O1, O2) and the webUIRequestUrl concordance links before drawing conclusions from a small difference.

Note also that ⁠rank_<label>_<measure>⁠ and ⁠percentile_rank_<label>_<measure>⁠ are computed within each label, over that label's own candidate set. Candidate sets usually differ in size between virtual corpora, so rank-based deltas compare positions in populations of different sizes.

Slow queries

Complex queries can take minutes on large corpora, especially on DeLiKo@DNB (⁠https://korap.dnb.de⁠), or exceed the time limit of the server, which leaves the results incomplete. As long as KorAP does not offer sampling by itself, a workaround is to restrict vc to a pseudo-random sample of texts, selected by the last digit of their sigle: "textSigle = /.*5/" selects about a tenth, "textSigle = /.*[0-4]/" about half of the texts. Combine the sample with other conditions using &, e.g. "textType = /.*Krimi.*/ & textSigle = /.*5/". Relative frequencies, and association scores that do not depend on the corpus size, such as logDice, remain estimates for the whole (virtual) corpus, whereas absolute frequencies, and with them log-likelihood scores, shrink with the sample. A sample saves about as much time as it leaves out if the server has recently read the index of the query terms, e.g. for the same query on another virtual corpus, but hardly any time otherwise, when most of the time goes into reading the index from disk. It thus pays off in series of queries on the same terms, such as a frequencyQuery() over many years or a collocationAnalysis(). Repeating a query on the same vc takes no time anyway, as its result comes from the local cache (see the cache parameter of KorAPConnection()). Queries that are fast anyway do not get faster, because selecting the sample takes some time itself.

See Also

Other collocation analysis functions: association-score-functions, collocationScoreQuery,KorAPConnection-method, synsemanticStopwords()

Examples

## Not run: 

# Find top collocates of "Packung" inside and outside the sports domain.
KorAPConnection(verbose = TRUE) |> auth() |>
  collocationAnalysis("Packung",
    vc = c("dmozDomain = /sport.*/", "dmozDomain != /sport.*/"),
    leftContextSize = 1, rightContextSize = 1, topCollocatesLimit = 20
  ) |>
  dplyr::filter(logDice >= 5)

## End(Not run)

## Not run: 

# Identify the most prominent light verb construction with "in ... setzen".
# Note that, currently, the use of focus function disallows exactFrequencies.
KorAPConnection(verbose = TRUE) |>
  collocationAnalysis("focus(in [tt/p=NN] {[tt/l=setzen]})",
    leftContextSize = 1, rightContextSize = 0, exactFrequencies = FALSE, topCollocatesLimit = 20
  )

## End(Not run)

Query frequencies of a node and a collocate and calculate collocation association scores

Description

Computes various collocation association scores based on frequencyQuery()s for a target word and a collocate.

Usage

## S4 method for signature 'KorAPConnection'
collocationScoreQuery(
  kco,
  node,
  collocate,
  vc = "",
  lemmatizeNodeQuery = FALSE,
  lemmatizeCollocateQuery = FALSE,
  leftContextSize = 5,
  rightContextSize = 5,
  scoreFunctions = defaultAssociationScoreFunctions(),
  smoothingConstant = 0.5,
  observed = NA,
  ignoreCollocateCase = FALSE,
  withinSpan = "base/s=s",
  cacheAs = NULL
)

Arguments

Value

tibble with query KorAP web request URL, all observed values and association scores

See Also

Other collocation analysis functions: association-score-functions, collocationAnalysis,KorAPConnection-method, synsemanticStopwords()

Examples

## Not run: 

KorAPConnection(verbose = TRUE) |>
  collocationScoreQuery("Grund", "triftiger")

## End(Not run)

## Not run: 

KorAPConnection(verbose = TRUE) |>
collocationScoreQuery("Grund", c("guter", "triftiger"),
   scoreFunctions = list(localMI = function(O1, O2, O, N, E, window_size) { O * log2(O/E) }) )

## End(Not run)

## Not run: 

library(highcharter)
library(tidyr)
KorAPConnection(verbose = TRUE) |>
  collocationScoreQuery("Team", "agil", vc = paste("pubDate in", c(2014:2018)),
                        lemmatizeNodeQuery = TRUE, lemmatizeCollocateQuery = TRUE) |>
                         pivot_longer(14:last_col(), names_to = "measure", values_to = "score") |>
  hchart(type="spline", hcaes(label, score, group=measure)) |>
  hc_add_onclick_korap_search()

## End(Not run)

Define corpus parts for a dispersion analysis

Description

Discover metadata groups and their token counts, or define a custom partition. Discovery checks the base corpus independently of query hits, so that parts without hits are retained without downloading the corpus.

Usage

corpusPartition(
  kco,
  by,
  values = NULL,
  vc = "",
  parts = NULL,
  classify = NULL,
  fields = NULL,
  verbose = kco@verbose,
  cacheAs = NULL,
  regex = NULL
)

Arguments

Value

A tibble of class KorAPCorpusPartition with part, value, vc, and tokens columns. Attributes record the metadata mapping, base corpus, server, index revision, covered tokens and excluded tokens. Overlapping parts are rejected by comparing their summed sizes with their union. An incomplete inventory explicitly restricts the population to its union, with a warning; it does not silently stand for the whole base corpus.

Examples

## Not run: 
kco <- KorAPConnection()
years <- corpusPartition(kco, "year", 2020:2022, vc = "referTo ratskorpus")
countries <- corpusPartition(kco, "region", c("DE", "AT", "CH"))
dispersionQuery(kco, "NutzerInnen", partitions = list(year = years))

## End(Not run)

Search corpus for query terms

Description

corpusQuery performs a corpus query via a connection to a KorAP-API-server

Usage

## S4 method for signature 'KorAPConnection'
corpusQuery(
  kco,
  query = if (missing(KorAPUrl)) {
    
    stop("At least one of the parameters query and KorAPUrl must be specified.", call. =
    FALSE)
 } else {
     httr2::url_parse(KorAPUrl)$query$q
 },
  vc = if (missing(KorAPUrl)) "" else httr2::url_parse(KorAPUrl)$query$cq,
  KorAPUrl,
  metadataOnly = TRUE,
  ql = if (missing(KorAPUrl)) "poliqarp" else httr2::url_parse(KorAPUrl)$query$ql,
  fields = c("corpusSigle", "corpusTitle", "textSigle", "pubDate", "pubPlace",
    "availability", "textClass", "dmozDomain", "wikiDomain", "snippet", "tokens"),
  accessRewriteFatal = TRUE,
  verbose = kco@verbose,
  expand = length(vc) != length(query),
  as.df = FALSE,
  context = NULL
)

Arguments

Value

Depending on the as.df parameter, a tibble or a KorAPQuery() object that, among other information, contains the total number of results in ⁠@totalResults⁠. Multiple queries return a summary table. With as.df = FALSE, it also retains query states for fetchAll(), fetchNext(), fetchSamplePages() and dispersionAnalysis(). Names of the query vector label the comparison columns. With as.df = TRUE, only the summary is retained. A corresponding URL to be used within a web browser is contained in ⁠@webUIRequestUrl⁠ Please make sure to check ⁠$collection$rewrites⁠ to see if any unforeseen access rewrites of the query's virtual corpus had to be performed.

Slow queries

Complex queries can take minutes on large corpora, especially on DeLiKo@DNB (⁠https://korap.dnb.de⁠), or exceed the time limit of the server, which leaves the results incomplete. As long as KorAP does not offer sampling by itself, a workaround is to restrict vc to a pseudo-random sample of texts, selected by the last digit of their sigle: "textSigle = /.*5/" selects about a tenth, "textSigle = /.*[0-4]/" about half of the texts. Combine the sample with other conditions using &, e.g. "textType = /.*Krimi.*/ & textSigle = /.*5/". Relative frequencies, and association scores that do not depend on the corpus size, such as logDice, remain estimates for the whole (virtual) corpus, whereas absolute frequencies, and with them log-likelihood scores, shrink with the sample. A sample saves about as much time as it leaves out if the server has recently read the index of the query terms, e.g. for the same query on another virtual corpus, but hardly any time otherwise, when most of the time goes into reading the index from disk. It thus pays off in series of queries on the same terms, such as a frequencyQuery() over many years or a collocationAnalysis(). Repeating a query on the same vc takes no time anyway, as its result comes from the local cache (see the cache parameter of KorAPConnection()). Queries that are fast anyway do not get faster, because selecting the sample takes some time itself.

References

https://ids-pub.bsz-bw.de/frontdoor/index/index/docId/9026

See Also

KorAPConnection(), fetchNext(), fetchRest(), fetchAll(), corpusStats()

Other corpus search functions: fetchAll,KorAPQuery-method, fetchAnnotations,KorAPQuery-method, fetchNext,KorAPQuery-method

Examples

## Not run: 

# Fetch basic metadata for "Ameisenplage"
KorAPConnection() |>
  corpusQuery("Ameisenplage") |>
  fetchAll()

# Fetch specific metadata fields for bibliographic analysis
query <- KorAPConnection() |>
  corpusQuery("Ameisenplage",
              fields = c("textSigle", "author", "title", "pubDate", "pubPlace", "textType"))
results <- fetchAll(query)
results@collectedMatches

## End(Not run)

## Not run: 

# Use the copy of a KorAP-web-frontend URL for an API query of "Ameise" in a virtual corpus
# and show the number of query hits (but don't fetch them).

KorAPConnection(verbose = TRUE) |>
  corpusQuery(
    KorAPUrl =
      "https://korap.ids-mannheim.de/?q=Ameise&cq=pubDate+since+2017&ql=poliqarp"
  )

## End(Not run)

## Not run: 

# Plot the time/frequency curve of "Ameisenplage"
KorAPConnection(verbose = TRUE) |>
  {
    . ->> kco
  } |>
  corpusQuery("Ameisenplage") |>
  fetchAll() |>
  slot("collectedMatches") |>
  mutate(year = lubridate::year(pubDate)) |>
  dplyr::select(year) |>
  group_by(year) |>
  summarise(Count = dplyr::n()) |>
  mutate(Freq = mapply(function(f, y) {
    f / corpusStats(kco, paste("pubDate in", y))@tokens
  }, Count, year)) |>
  dplyr::select(-Count) |>
  complete(year = min(year):max(year), fill = list(Freq = 0)) |>
  plot(type = "l")

## End(Not run)

Get corpus size and statistics

Description

Retrieve information about corpus size (documents, tokens, sentences, paragraphs) for the entire corpus or a virtual corpus subset.

Usage

## S4 method for signature 'KorAPConnection'
corpusStats(kco, vc = "", verbose = kco@verbose, as.df = FALSE, cacheAs = NULL)

Arguments

Value

Object containing corpus statistics with the following information:

vc

Virtual corpus definition used (empty string for entire corpus)

documents

Total number of documents in the (virtual) corpus

tokens

Total number of word tokens in the (virtual) corpus

sentences

Total number of sentences in the (virtual) corpus

paragraphs

Total number of paragraphs in the (virtual) corpus

webUIRequestUrl

URL to view this corpus subset in KorAP web interface

When as.df=TRUE, returns a data frame with these columns. When as.df=FALSE (default), returns a KorAPCorpusStats object with these values as slots.

Usage

# Get statistics for entire corpus
kcon <- KorAPConnection()
stats <- corpusStats(kcon)

# Get statistics for a specific time period
stats <- corpusStats(kcon, "pubDate in 2020")

# Access the number of tokens
stats@tokens

Examples

## Not run: 

kco <- KorAPConnection()

# Get statistics for entire corpus (returns S4 object)
stats <- corpusStats(kco)
stats@tokens  # Access number of tokens

# Get statistics for newspaper texts from 2017 (as data frame)
df <- corpusStats(kco, "pubDate in 2017 & textType=/Zeitung.*/", as.df = TRUE)
df$documents  # Access number of documents

# Compare corpus sizes across years
years <- 2015:2020
sizes <- sapply(years, function(y) {
  corpusStats(kco, paste("pubDate in", y))@tokens
})

## End(Not run)

Default KorAP server URL

Description

Returns the KorAP instance URL to connect to if none is given explicitly: the environment variable KORAP_URL if it is set and non-empty, and the IDS Mannheim KorAP main instance otherwise.

Usage

defaultKorAPUrl()

Value

URL of the KorAP instance to connect to by default

Dispersion measures and user-defined scores

Description

Usage

defaultDispersionFunctions()

## S4 method for signature 'numeric'
dispersionAnalysis(
  frequencies,
  sizes,
  scoreFunctions = defaultDispersionFunctions()
)

dispersionDP(frequencies, sizes)

dispersionDPnorm(frequencies, sizes)

dispersionJuillandD(frequencies, sizes)

dispersionCarrollD2(frequencies, sizes)

dispersionRosengrenS(frequencies, sizes)

dispersionRange(frequencies, sizes)

Arguments

Details

Calculate dispersion from absolute frequencies and corpus-part sizes. The parts must be disjoint; include positive-sized parts without hits. Sizes are token counts (or proportional sizes), not the frequencies of the query. The analysis interface and result structure may change in future releases.

dispersionDP() computes sum(abs(frequencies / sum(frequencies) - sizes / sum(sizes))) / 2. dispersionDPnorm() divides DP by 1 - min(sizes / sum(sizes)), following the corrected normalization in Lijffijt and Gries (2012). Smaller DP and DPnorm mean more even distribution. Juilland's D, Carroll's D2 and Rosengren's S use the versions for unequal part sizes described by Gries (2020); larger values mean more even distribution. These are independent implementations of the published formulas; Gries' companion script is not included or required. dispersionRange() counts parts with hits, not documents unless documents are the parts.

The built-in measures other than range return NA_real_ if there are no hits or fewer than two positive-sized parts. Range is zero without hits. No measure is assumed to be appropriate for every research question, and the measures are not combined into a rank or adjusted frequency.

Value

dispersionAnalysis() returns a one-row tibble, one column per score. defaultDispersionFunctions() returns a named list of functions. Individual score functions return a numeric scalar.

Interpretation

Range describes observed reach: the number of parts containing at least one hit. It ignores part sizes and within-part frequencies, and depends on the total number of hits. Broader reach need not mean more even distribution.

DP compares each part's share of hits with its share of corpus tokens. For example, DP = 0.20 means that 20 percent of the hit distribution would have to be reallocated between parts to match the token shares, keeping the total fixed. This is hypothetical hit mass, allowing fractional counts; it is not a percentage of sources or speakers. Smaller is closer, and zero is an exact match. DPnorm rescales DP by its maximum for the given part sizes; the redistribution interpretation applies to DP itself, not directly to DPnorm.

Juilland's D describes variation in relative hit frequencies across parts; Carroll's D2 describes their entropy. Both equal one when each part has the same relative frequency. Rosengren's S compares the hit and token distributions, reaching one when they match. Larger D, D2 and S mean more even distribution under their respective definitions. The measures emphasize different aspects, so they need not rank variants in the same order.

Compare variants within the same corpus and partition. There is no universal cutoff for good dispersion, and these descriptive scores are not significance tests. Averaging ranks would depend on the selected measures, give related measures repeated votes and discard the size of differences. Inspect the underlying part counts and token sizes, and report disagreements explicitly. The leading one or two sources' shares of hits, considered alongside their token shares, can help explain concentration. Concordances are needed to investigate possible house styles, quotation or syndicated material.

References

Gries, Stefan Th. (2008). Dispersions and adjusted frequencies in corpora. International Journal of Corpus Linguistics, 13(4), 403-437. doi:10.1075/ijcl.13.4.02gri.

Gries, Stefan Th. (2020). Analyzing dispersion. In M. Paquot & S. Th. Gries (eds.), A Practical Handbook of Corpus Linguistics, 99-118. doi:10.1007/978-3-030-46216-1_5. Companion resources: https://www.stgries.info/research/dispersion/links.html.

Lijffijt, Jefrey & Gries, Stefan Th. (2012). Correction to "Dispersions and adjusted frequencies in corpora". International Journal of Corpus Linguistics, 17(1), 147-149. doi:10.1075/ijcl.17.1.08lij.

Juilland, Alphonse G. & Chang-Rodriguez, Eugenio (1964). Frequency dictionary of Spanish words. The Hague: Mouton.

Carroll, John B. (1970). An alternative to Juilland's usage coefficient for lexical frequencies, and a proposal for a standard frequency index (SFI). Computer Studies in the Humanities and Verbal Behavior, 3(2), 61-65.

Rosengren, Inger (1971). The quantitative concept of language and its relation to the structure of frequency dictionaries. Etudes de linguistique appliquee (Nouvelle Serie), 1, 103-127.

Examples

dispersionAnalysis(c(1, 2, 3, 4, 5), c(9, 10, 10, 10, 11))
dispersionAnalysis(c(10, 0), c(100, 100), scoreFunctions = list(
  DP = dispersionDP,
  evenness = function(frequencies, sizes) 1 - dispersionDP(frequencies, sizes)
))

Calculate dispersion directly from a corpus query

Description

Usage

## S4 method for signature 'KorAPQuery'
dispersionAnalysis(
  frequencies,
  by = "source",
  regex = NULL,
  scoreFunctions = defaultDispersionFunctions(),
  exactFrequencies = NULL,
  searchHitsSampleLimit = 20000,
  seed = 7,
  verbose = frequencies@korapConnection@verbose,
  cacheAs = NULL
)

## S4 method for signature 'data.frame'
dispersionAnalysis(
  frequencies,
  by = "source",
  regex = NULL,
  scoreFunctions = defaultDispersionFunctions(),
  exactFrequencies = NULL,
  searchHitsSampleLimit = 20000,
  seed = 7,
  verbose = NULL,
  cacheAs = NULL
)

Arguments

Details

Ask a corpus question first, then choose the metadata dimension of its dispersion. The library discovers the groups and their sizes, including groups without query hits. No source lists or classification code are needed. The analysis interface and result structure may change in future releases.

An explicit fetchSamplePages() step makes the retrieval scope visible. Recorded pages are reused without drawing again; any missing metadata are retrieved from those same pages. Multiple queries share independently discovered groups and token sizes, seeded by the groups already observed across their fetched metadata. Printing their result shows one column per query variant, labelled by the names of the original query vector.

See the Interpretation section in dispersion-score-functions for what the measures describe, their directions and why they can disagree. Use distribution to examine source counts and token sizes alongside the scores.

by = "year" treats publication years as discrete corpus parts, using each year's token count as its size and retaining positive-sized years without query hits. The built-in measures ignore temporal order and distances; they describe dispersion across year groups, not a trend over time.

Use a metadata-only query (the default of corpusQuery()). A full-text query rewritten to a smaller licensed population is rejected, so that hit counts and corpus-part sizes cannot silently refer to different populations.

All metadata categories are discovered independently of the query hits. The current API does not offer a public distinct-values endpoint, so the library retrieves one representative text's metadata from each remaining group, excluding already known groups until none remain. It does not download the corpus. The resulting partition is cached per server, index revision, virtual corpus, metadata field and grouping rule when connection caching is enabled. Discovery and size queries can take time on the first call, especially for fields with many categories.

regex removes a suffix, rather than selecting which categories are included. It must have a corresponding corpus-query regex: use ordinary groups and character classes, without lookarounds. This makes the metadata grouping and the token denominators agree. Advanced arbitrary groupings remain available through corpusPartition() and dispersionQuery().

Value

A KorAPDispersion tibble. Printing it shows all requested scores. Detailed counts remain in distribution; sampling records whether the scores are estimated. The complete partitions are also retained as the partitions attribute. See dispersionQuery() for the other columns.

Examples

## Not run: 
kco <- KorAPConnection()
q <- corpusQuery(kco, "NutzerInnen", vc = "pubPlaceKey=AT & pubDate in 2022")
dispersionAnalysis(q, by = "source")
# The explicit metadata-field equivalent:
dispersionAnalysis(q, by = "corpusTitle", regex = " [0-9]{4}$")

## End(Not run)

Query dispersion across corpus metadata

Description

Compute selectable dispersion measures for one or more queries, using exact counts per part or a reproducible metadata sample shared by all partitions.

Usage

## S4 method for signature 'KorAPConnection'
dispersionQuery(
  kco,
  query,
  partitions = NULL,
  scoreFunctions = defaultDispersionFunctions(),
  exactFrequencies = TRUE,
  searchHitsSampleLimit = 20000,
  seed = 7,
  ql = "poliqarp",
  verbose = kco@verbose,
  cacheAs = NULL,
  by = "source",
  regex = NULL,
  vc = ""
)

Arguments

Details

A sample is a simple random sample of result pages, not independent hits. Every hit has the same page-inclusion probability. The returned distribution retains unobserved parts and reports observed and extrapolated hit counts. Ratios of these estimates provide plug-in estimates of dispersion; nonlinear measures can be strongly biased by within-page clustering, even with many sampled hits. Range from a sample is only a lower bound, not an estimate of the number of occupied parts. No confidence bounds are claimed. Missing or unlisted metadata are counted as excluded hits.

Partition sizes come from independent corpus statistics, never from the sample. Failed, incomplete or timed-out requests abort the analysis rather than becoming zeros. exactFrequencies = TRUE never falls back to sampling. Arbitrary parts require a membership classifier for metadata sampling. Reuse a partition only with its original base corpus, server and index revision.

Value

A tibble with one row per query and partition, the requested scores, totalResults (base-corpus hits), sampledHits (metadata rows analyzed), usedHits, excludedHits, nParts, estimated, rangeIsLowerBound, and list columns distribution and sampling. Distributions include corpus-part sizes, observed counts, estimated frequencies, relative frequencies and concordance URLs. sampling records the method, seed, selected pages, inclusion probability, server and index revision.

Examples

## Not run: 
kco <- KorAPConnection()
years <- corpusPartition(kco, "year", 2020:2022)
dispersionQuery(kco, "NutzerInnen", years, scoreFunctions = list(
  evenness = function(frequencies, sizes) 1 - dispersionDP(frequencies, sizes)
))
dispersionQuery(kco, "NutzerInnen", years, exactFrequencies = FALSE,
                searchHitsSampleLimit = 10000, seed = 7)

## End(Not run)

Fetch all results of a KorAP query.

Description

fetchAll fetches all results of a KorAP query.

Usage

## S4 method for signature 'KorAPQuery'
fetchAll(kqo, verbose = kqo@korapConnection@verbose, ...)

## S4 method for signature 'data.frame'
fetchAll(kqo, verbose = NULL, ...)

Arguments

Value

The updated kqo object with all results in ⁠@collectedMatches⁠

See Also

Other corpus search functions: corpusQuery,KorAPConnection-method, fetchAnnotations,KorAPQuery-method, fetchNext,KorAPQuery-method

Examples

## Not run: 
# Fetch all metadata of every query hit for "Ameisenplage" and show a summary
q <- KorAPConnection() |>
  corpusQuery("Ameisenplage") |>
  fetchAll()
q@collectedMatches

# Fetch also all KWICs
q <- KorAPConnection() |> auth() |>
 corpusQuery("Ameisenplage", metadataOnly = FALSE) |>
 fetchAll()
q@collectedMatches

# Retrieve title and text sigle metadata of all texts published on 1958-03-12
q <- KorAPConnection() |>
 corpusQuery("<base/s=t>", # this matches each text once
    vc = "pubDate in 1958-03-12",
    fields = c("textSigle", "title")
) |>
 fetchAll()
q@collectedMatches

## End(Not run)

Fetch annotations (generic)

Description

S4 generic for fetching token annotations for collected matches. See specific methods, e.g. fetchAnnotations,KorAPQuery-method.

Usage

fetchAnnotations(kqo, foundry = "tt", overwrite = FALSE,
  verbose = kqo@korapConnection@verbose)

Arguments

See Also

fetchAnnotations,KorAPQuery-method

Fetch annotations for all collected matches

Description

Usage

## S4 method for signature 'KorAPQuery'
fetchAnnotations(
  kqo,
  foundry = "tt",
  overwrite = FALSE,
  verbose = kqo@korapConnection@verbose
)

Arguments

Details

fetchAnnotations fetches annotations (only token annotations, for now) for all matches in the ⁠@collectedMatches⁠ slot of a KorAPQuery object and adds annotation columns directly to the ⁠@collectedMatches⁠ data frame. The method uses the matchID from collected matches.

Important: For copyright-restricted corpora, users must be authorized via auth() and the initial corpus query must have metadataOnly = FALSE to ensure snippets are available for annotation parsing.

The method parses XML snippet annotations and adds linguistic columns to the data frame:

• pos: data frame with left, match, right columns, each containing list vectors of part-of-speech tags

• lemma: data frame with left, match, right columns, each containing list vectors of lemmas

• morph: data frame with left, match, right columns, each containing list vectors of morphological tags

• atokens: data frame with left, match, right columns, each containing list vectors of token text (from annotations)

• annotation_snippet: original XML snippet from the annotation API

Value

The updated kqo object with annotation columns like pos, lemma, morph (and atokens and annotation_snippet) in the ⁠@collectedMatches⁠ slot. Each column is a data frame with left, match, and right columns containing list vectors of annotations for the left context, matched tokens, and right context, respectively. The original XML snippet for each match is also stored in annotation_snippet.

See Also

Other corpus search functions: corpusQuery,KorAPConnection-method, fetchAll,KorAPQuery-method, fetchNext,KorAPQuery-method

Examples

## Not run: 

# Fetch annotations for matches using TreeTagger foundry
# Note: Authorization required for copyright-restricted corpora
q <- KorAPConnection() |>
  auth() |>
  corpusQuery("Ameisenplage", metadataOnly = FALSE) |>
  fetchNext(maxFetch = 10) |>
  fetchAnnotations()

# Access linguistic annotations for match i:
pos_tags <- q@collectedMatches$pos
# Data frame with left/match/right columns for POS tags
lemmas <- q@collectedMatches$lemma
# Data frame with left/match/right columns for lemmas
morphology <- q@collectedMatches$morph
# Data frame with left/match/right columns for morphological tags
atokens <- q@collectedMatches$atokens
# Data frame with left/match/right columns for annotation token text
# Original XML snippet for match i
raw_snippet <- q@collectedMatches$annotation_snippet[[i]]

# Access specific components:
# POS tags for the matched tokens in match i
match_pos <- q@collectedMatches$pos$match[[i]]
# Lemmas for the left context in match i
left_lemmas <- q@collectedMatches$lemma$left[[i]]
# Token text for the right context in match i
right_tokens <- q@collectedMatches$atokens$right[[i]]

# Use a different foundry (e.g., MarMoT)
q <- KorAPConnection() |>
  auth() |>
  corpusQuery("Ameisenplage", metadataOnly = FALSE) |>
  fetchNext(maxFetch = 10) |>
  fetchAnnotations(foundry = "marmot")
q@collectedMatches$pos$left[1] # POS tags for the left context of the first match

## End(Not run)

Fetch the next bunch of results of a KorAP query.

Description

fetchNext fetches the next bunch of results of a KorAP query.

Usage

## S4 method for signature 'KorAPQuery'
fetchNext(
  kqo,
  offset = kqo@nextStartIndex,
  maxFetch = maxResultsPerPage,
  verbose = kqo@korapConnection@verbose,
  randomizePageOrder = FALSE,
  preserveMetadataValues = NULL
)

## S4 method for signature 'data.frame'
fetchNext(kqo, ..., verbose = NULL)

Arguments

Value

The kqo input object with updated slots collectedMatches, apiResponse, nextStartIndex, hasMoreMatches With preserveMetadataValues = TRUE, metadata columns with at most one value per match are atomic vectors (missing values become NA). Truly multivalued columns remain lists; a later page may introduce multiple values and promote a previously scalar column to a list.

References

https://ids-pub.bsz-bw.de/frontdoor/index/index/docId/9026

See Also

Other corpus search functions: corpusQuery,KorAPConnection-method, fetchAll,KorAPQuery-method, fetchAnnotations,KorAPQuery-method

Examples

## Not run: 

q <- KorAPConnection() |>
  corpusQuery("Ameisenplage") |>
  fetchNext()
q@collectedMatches

## End(Not run)

Fetches the remaining results of a KorAP query.

Description

Fetches the remaining results of a KorAP query.

Usage

## S4 method for signature 'KorAPQuery'
fetchRest(kqo, verbose = kqo@korapConnection@verbose, ...)

## S4 method for signature 'data.frame'
fetchRest(kqo, verbose = NULL, ...)

Arguments

Value

The updated kqo object with remaining results in ⁠@collectedMatches⁠

Examples

## Not run: 

q <- KorAPConnection() |>
  corpusQuery("Ameisenplage") |>
  fetchRest()
q@collectedMatches

## End(Not run)

Fetch a reproducible sample of result pages

Description

Select complete result pages without replacement. This is a page (cluster) sample, not a simple random sample of individual hits. Hits within a selected page remain in Lucene index order. A single randomly selected page therefore does not provide independently sampled hits. If all results fit within the limit, all pages are fetched in index order and no sample is drawn.

Usage

## S4 method for signature 'KorAPQuery'
fetchSamplePages(
  kqo,
  maxHits = 2000,
  seed = 7,
  verbose = kqo@korapConnection@verbose,
  warn = TRUE
)

## S4 method for signature 'data.frame'
fetchSamplePages(kqo, maxHits = 2000, seed = 7, verbose = NULL, warn = TRUE)

Arguments

Details

Unlike taking the first hits, pages are selected from the entire result set. If several pages are available, even one page can be selected at random. Within-page ordering and clustering remain; this distinction matters when interpreting estimates and uncertainty. The actual row count can be below maxHits, particularly when the last, shorter page is selected.

Call fetchAll() to replace a page sample with the complete result. A page sample is not a sequential prefix, so fetchNext() and fetchRest() cannot safely continue it. Call fetchSamplePages() again to draw a new sample.

Value

Updated query object(s), with hits in collectedMatches and page sampling provenance in the korapSampling attribute. dispersionAnalysis() uses this exact sample and its inclusion probability without drawing again. Single-valued metadata columns are atomic vectors, with NA for missing values; genuinely multivalued columns remain lists.

Examples

## Not run: 
KorAPConnection() |>
  corpusQuery("Gendern") |>
  fetchSamplePages(maxHits = 2000, seed = 7) |>
  dispersionAnalysis(by = "source")

## End(Not run)

Format duration in seconds to human-readable format

Description

Converts a duration in seconds to a formatted string with days, hours, minutes, and seconds. Used for ETA calculations and progress reporting.

Usage

format_duration(seconds)

Arguments

Value

character string with formatted duration

Examples

## Not run: 
format_duration(3661) # "01h 01m 01s"
format_duration(86461) # "1d 00h 01m 01s"

## End(Not run)

Query frequencies of search expressions in virtual corpora

Description

frequencyQuery combines corpusQuery(), corpusStats() and ci() to compute a tibble with the absolute and relative frequencies and confidence intervals of one or multiple search terms across one or multiple virtual corpora.

Usage

## S4 method for signature 'KorAPConnection'
frequencyQuery(
  kco,
  query,
  vc = "",
  conf.level = 0.95,
  as.alternatives = FALSE,
  cacheAs = NULL,
  dispersionBy = NULL,
  dispersionFunctions = defaultDispersionFunctions(),
  dispersionExactFrequencies = TRUE,
  searchHitsSampleLimit = 20000,
  seed = 7,
  ...
)

Arguments

Value

A tibble, with each row containing the following result columns for query and vc combinations:

• query: the query string used for the frequency analysis.

• variant: query-vector names, when supplied, for display labels.

• as.alternatives: denominator mode, retained for plotting helpers.

• totalResults: absolute frequency of query matches in the vc.

• vc: virtual corpus used for the query.

• queryDuration: client-side duration of the request in seconds.

• webUIRequestUrl: URL of the corresponding web UI request with respect to query and vc.

• total: total number of words in vc.

• f: relative frequency of query matches in the vc.

• conf.low: lower bound of the confidence interval for the relative frequency, given conf.level.

• conf.high: upper bound of the confidence interval for the relative frequency, given conf.level.

Slow queries

Complex queries can take minutes on large corpora, especially on DeLiKo@DNB (⁠https://korap.dnb.de⁠), or exceed the time limit of the server, which leaves the results incomplete. As long as KorAP does not offer sampling by itself, a workaround is to restrict vc to a pseudo-random sample of texts, selected by the last digit of their sigle: "textSigle = /.*5/" selects about a tenth, "textSigle = /.*[0-4]/" about half of the texts. Combine the sample with other conditions using &, e.g. "textType = /.*Krimi.*/ & textSigle = /.*5/". Relative frequencies, and association scores that do not depend on the corpus size, such as logDice, remain estimates for the whole (virtual) corpus, whereas absolute frequencies, and with them log-likelihood scores, shrink with the sample. A sample saves about as much time as it leaves out if the server has recently read the index of the query terms, e.g. for the same query on another virtual corpus, but hardly any time otherwise, when most of the time goes into reading the index from disk. It thus pays off in series of queries on the same terms, such as a frequencyQuery() over many years or a collocationAnalysis(). Repeating a query on the same vc takes no time anyway, as its result comes from the local cache (see the cache parameter of KorAPConnection()). Queries that are fast anyway do not get faster, because selecting the sample takes some time itself.

Examples

## Not run: 

KorAPConnection(verbose = TRUE) |>
  frequencyQuery(c("Mücke", "Schnake"), paste0("pubDate in ", 2000:2003))

## End(Not run)

Plot frequency bars with confidence intervals using ggplot2

Description

Add a column layer and aligned confidence-interval error bars to a ggplot. Like geom_freq_by_year_ci(), this helper uses the values already mapped in the plot: convert a frequencyQuery() result with ipm() first, or with percent() for alternatives. It does not query KorAP or rescale the data.

Usage

geom_freq_bar_ci(
  mapping = NULL,
  data = NULL,
  position = "dodge",
  width = 0.9,
  errorbar.width = 0.25,
  errorbar.colour = NULL,
  errorbar.params = list(),
  na.rm = FALSE,
  orientation = "x",
  showCI = TRUE,
  ...
)

Arguments

Details

Map fill or group to a query or corpus variable for grouped comparisons. Map group explicitly when fixed fill or colour settings override a grouping aesthetic, so both layers retain the same groups. Stacked bars are not supported because independently calculated confidence limits cannot be stacked. Missing observations are not converted to zeros; use explicit missing rows to reserve slots for absent query/corpus pairs.

Use orientation = "y" with aes(x = ipm, y = label), or use ggplot2::coord_flip() with the default mapping for horizontal bars. Themes, facets, scales and ggplot2::ggsave() work as usual, including PDF and SVG output.

Value

A list of ggplot2 layers, added to a plot with +.

See Also

hc_freq_bar_ci() for interactive charts; geom_freq_by_year_ci() for time series.

Examples

## Not run: 
library(ggplot2)
kco <- KorAPConnection("https://korap.dnb.de")
frequencyQuery(kco, "Berlin",
  vc = c(Buchpreis = "award=buchpreis", DeLiKo = "")) |>
  ipm() |>
  ggplot(aes(x = label, y = ipm)) +
  geom_freq_bar_ci() +
  theme_bw()

frequencyQuery(kco, c("Berlin", "Paris"),
  vc = c(Buchpreis = "award=buchpreis", DeLiKo = ""), expand = TRUE) |>
  ipm() |>
  ggplot(aes(x = label, y = ipm, fill = query)) +
  geom_freq_bar_ci() +
  coord_flip()

## End(Not run)

Add KorAP search click events to highchart plots

Description

Adds on-click events to data points of highcharts that were constructed with frequencyQuery() or collocationScoreQuery(). Clicks on data points then launch KorAP web UI queries for the given query term and virtual corpus in a separate tab that is reused for subsequent clicks. Only HTTP(S) links are opened; points without a valid link are not clickable.

Usage

hc_add_onclick_korap_search(hc)

Arguments

Value

The input highchart object with added on-click events.

See Also

Other highcharter-helpers: hc_dispersion(), hc_freq_bar_ci(), hc_freq_by_year_ci()

Examples

## Not run: 

library(highcharter)
library(tidyr)

KorAPConnection(verbose = TRUE) %>%
  collocationScoreQuery("Team", "agil", vc = paste("pubDate in", c(2014:2018)),
                        lemmatizeNodeQuery = TRUE, lemmatizeCollocateQuery = TRUE) %>%
                         pivot_longer(c("O", "E")) %>%
  hchart(type="spline", hcaes(label, value, group=name)) %>%
  hc_add_onclick_korap_search()

## End(Not run)

Plot interactive distributions underlying a dispersion analysis

Description

Compare each query's share of hits with the corpus-token share of each part. The chart uses stored distributions and makes no requests to KorAP.

Usage

hc_dispersion(df, partition = NULL, top = NULL)

Arguments

Details

Percentages use hits and tokens within the listed, positive-sized parts, matching the population used by the dispersion measures. Positive-sized parts without hits are retained. A zero-hit variant has no defined hit-share distribution and is identified in the caption without drawing hit-share bars.

Grey bars show corpus-token shares; coloured bars show the query variants. Tooltips distinguish shares of total frequency from relative frequencies per million tokens, and show absolute frequencies and subcorpus sizes. Click individual variant bars to open their stored concordance links. Corpus-token bars and combined remainder bars have no concordance link.

Sample-based results are labelled as estimates, with inclusion probabilities in tooltips. Page clustering can strongly bias these shares. The chart does not provide confidence intervals or an overall ranking of dispersion scores. The interface and display may change in future releases.

Value

A highchart htmlwidget, customizable with highcharter functions.

Display language

Generated chart text and number formatting follow the browser's preferred German or English language, falling back to English. Query names and custom labels are retained. Set options(rkorap.chart.language = "de") or "en" before creating charts to fix the language for exports; "auto" restores browser selection. IPM means instances per million tokens.

See Also

Other highcharter-helpers: hc_add_onclick_korap_search(), hc_freq_bar_ci(), hc_freq_by_year_ci()

Examples

## Not run: 
d <- KorAPConnection() |>
  corpusQuery(c(star = "[orth='Nutzer*innen']",
                colon = "[orth='Nutzer:innen']")) |>
  fetchSamplePages(maxHits = Inf) |>
  dispersionAnalysis(by = "source")
hc_dispersion(d, top = 12)

## End(Not run)

Plot interactive frequency bars with confidence intervals

Description

Compare relative frequencies from a frequencyQuery() result. Frequencies and confidence limits are converted to instances per million tokens, or to percentages for alternatives. The function uses the stored results without making further requests to KorAP.

Usage

hc_freq_bar_ci(
  df,
  as.alternatives = NULL,
  ylabel = if (as.alternatives) "%" else "IPM",
  horizontal = TRUE,
  log = FALSE,
  showCI = TRUE,
  ...
)

Arguments

Details

A single query is compared across virtual corpora; multiple queries in one corpus are compared across queries. With multiple queries and corpora, each corpus is a category with one bar per query. Categories and query groups keep their first-appearance order. A label column, supplied by named virtual corpora, is used for corpus labels.

Each query/corpus pair must occur at most once. Missing combinations and rows without a frequency or a positive denominator appear as gaps, not as zero counts. Zero hits in a non-empty corpus remain visible with their confidence interval. Error bars use the confidence limits supplied in df, including the confidence level chosen in frequencyQuery().

Tooltips show absolute and relative frequencies, corpus size (or the total frequency of alternatives) and confidence intervals. Clicking a bar opens its stored KorAP concordance link, when available.

Value

A highchart htmlwidget, customizable with highcharter functions.

Display language

Generated chart text and number formatting follow the browser's preferred German or English language, falling back to English. Query names and custom labels are retained. Set options(rkorap.chart.language = "de") or "en" before creating charts to fix the language for exports; "auto" restores browser selection. IPM means instances per million tokens.

See Also

geom_freq_bar_ci() for ggplot2 figures suitable for print.

Other highcharter-helpers: hc_add_onclick_korap_search(), hc_dispersion(), hc_freq_by_year_ci()

Examples

## Not run: 
kco <- KorAPConnection("https://korap.dnb.de")
frequencyQuery(kco, "Berlin",
  vc = c(Buchpreis = "award=buchpreis", DeLiKo = "")) |>
  hc_freq_bar_ci()

frequencyQuery(kco, c("Iowa", "Berlin"), vc = "award=buchpreis") |>
  hc_freq_bar_ci(horizontal = FALSE)

frequencyQuery(kco, c("Berlin", "Paris"),
  vc = c(Buchpreis = "award=buchpreis", DeLiKo = ""), expand = TRUE) |>
  hc_freq_bar_ci()

## End(Not run)

Plot interactive frequency curves with confidence intervals

Description

Convenience function for plotting typical frequency by year graphs with confidence intervals using highcharter.

Warning: This function may be moved to a new package.

Usage

hc_freq_by_year_ci(
  df,
  as.alternatives = NULL,
  ylabel = if (as.alternatives) "%" else "IPM",
  smooth = FALSE,
  showCI = TRUE,
  ...
)

Arguments

Value

A highchart htmlwidget object containing the frequency plot.

Display language

Generated chart text and number formatting follow the browser's preferred German or English language, falling back to English. Query names and custom labels are retained. Set options(rkorap.chart.language = "de") or "en" before creating charts to fix the language for exports; "auto" restores browser selection. IPM means instances per million tokens.

See Also

Other highcharter-helpers: hc_add_onclick_korap_search(), hc_dispersion(), hc_freq_bar_ci()

Examples

## Not run: 

year <- c(1990:2018)
alternatives <- c("macht []{0,3} Sinn", "ergibt []{0,3} Sinn")
KorAPConnection(verbose = TRUE) %>%
  frequencyQuery(query = alternatives,
                 vc = paste("textType = /Zeit.*/ & pubDate in", year),
                 as.alternatives = TRUE) %>%
  hc_freq_by_year_ci(as.alternatives = TRUE)


kco <- KorAPConnection(verbose = TRUE)
expand_grid(
  condition = c("idsColumn = /Wirtschaft.*/", "idsColumn != /Wirtschaft.*/"),
  year = (2005:2011)
) %>%
  cbind(frequencyQuery(
    kco,
    "[tt/l=Heuschrecke]",
    paste0(.$condition, " & pubDate in ", .$year)
  ))  %>%
  hc_freq_by_year_ci()

## End(Not run)

Initialize KorAPConnection object

Description

Initialize KorAPConnection object

Usage

## S4 method for signature 'KorAPConnection'
initialize(
  .Object,
  KorAPUrl = defaultKorAPUrl(),
  apiVersion = "v1.0",
  apiUrl,
  accessToken = getAccessToken(KorAPUrl),
  oauthClient = NULL,
  oauthScope = "search match_info",
  authorizationSupported = TRUE,
  userAgent = "R-KorAP-Client",
  timeout = 240,
  verbose = FALSE,
  cache = TRUE
)

Initialize KorAPQuery object

Description

Initialize KorAPQuery object

Usage

## S4 method for signature 'KorAPQuery'
initialize(
  .Object,
  korapConnection = NULL,
  request = NULL,
  vc = "",
  totalResults = 0,
  nextStartIndex = 0,
  fields = c("corpusSigle", "corpusTitle", "textSigle", "pubDate", "pubPlace",
    "availability", "textClass", "dmozDomain", "wikiDomain", "snippet", "tokens"),
  requestUrl = "",
  webUIRequestUrl = "",
  apiResponse = NULL,
  hasMoreMatches = FALSE,
  collectedMatches = NULL
)

Arguments

Logging utilities for RKorAPClient

Description

This module provides centralized logging functions used throughout the package for progress reporting and ETA calculations. Log informational messages with optional coloring

Usage

log_info(v, ...)

Arguments

Merge duplicate collocate rows and re-calculate association scores and URLs. Useful if collocation analyses were performed separately for collocates on the left and right side of a node.

Description

Merge duplicate collocate rows and re-calculate association scores and URLs. Useful if collocation analyses were performed separately for collocates on the left and right side of a node.

Usage

mergeDuplicateCollocates(..., smoothingConstant = 0.5)

Arguments

Value

tibble with unique collocate rows

Parse XML annotations into linguistic layers

Description

Internal helper function to extract linguistic annotations (lemma, POS, morphology) from XML annotation snippets returned by the KorAP API.

Usage

parse_xml_annotations(xml_snippet)

Arguments

Value

Named list with vectors for 'token', 'lemma', 'pos', and 'morph'

Parse XML annotations into linguistic layers with left/match/right structure

Description

Internal helper function to extract linguistic annotations (lemma, POS, morphology) from XML annotation snippets returned by the KorAP API, split into left context, match, and right context sections like the tokens field.

Usage

parse_xml_annotations_structured(xml_snippet)

Arguments

Value

Named list with nested structure containing left/match/right for 'atokens', 'lemma', 'pos', and 'morph'

Persist current access token in keyring

Description

Persist current access token in keyring

Usage

## S4 method for signature 'KorAPConnection'
persistAccessToken(kco, accessToken = kco@accessToken)

Arguments

Value

KorAPConnection object.

See Also

clearAccessToken(), auth()

Other initialization functions: KorAPConnection-class, auth,KorAPConnection-method, clearAccessToken,KorAPConnection-method

Examples

## Not run: 
kco <- KorAPConnection(accessToken = "e739u6eOzkwADQPdVChxFg")
persistAccessToken(kco)

kco <- KorAPConnection() %>%
  auth(app_id = "<my application id>") %>%
  persistAccessToken()

## End(Not run)

Print dispersion measures without hiding them behind metadata columns

Description

Print dispersion measures without hiding them behind metadata columns

Usage

## S3 method for class 'KorAPDispersion'
print(x, ..., width = Inf, n = Inf)

Arguments

Value

The input, invisibly.

Objects exported from other packages

Description

These objects are imported from other packages. Follow the links below to see their documentation.

broom

tidy

dplyr

bind_cols(), group_by(), mutate(), n(), select(), summarise()

lubridate

year()

magrittr

%>%

tibble

as_tibble()

tidyr

complete(), expand_grid()

Display KorAPConnection object

Description

Display KorAPConnection object

Usage

## S4 method for signature 'KorAPConnection'
show(object)

Arguments

Preliminary synsemantic stopwords function

Description

Preliminary synsemantic stopwords function to be used in collocation analysis.

Usage

synsemanticStopwords(...)

Arguments

Details

Currently only suitable for German. See stopwords package for other languages.

Value

Vector of synsemantic stopwords.

See Also

Other collocation analysis functions: association-score-functions, collocationAnalysis,KorAPConnection-method, collocationScoreQuery,KorAPConnection-method

Retrieve metadata for a text, identified by its sigle (id)

Description

Retrieves metadata for a text, identified by its sigle (id) using the corresponding KorAP API (see Kustvakt Wiki). To retrieve the metadata for every text in a virtual corpus, use corpusQuery() with ⁠<base/s=t>⁠ as query, instead.

Usage

## S4 method for signature 'KorAPConnection'
textMetadata(kco, textSigle, verbose = kco@verbose, cacheAs = NULL)

Arguments

Value

Tibble with columns for each metadata property. In case of errors, such as non-existing texts/sigles, the tibble will also contain a column called errors. If there are metadata columns you cannot make sense of, please ignore them. The function simply returns all the metadata it gets from the server.

Examples

## Not run: 
KorAPConnection() |> textMetadata(c("WUD17/A97/08542", "WUD17/B96/57558", "WUD17/A97/08541"))

## End(Not run)

Take cached results as they are for the duration of an expression

Description

Sets the cacheAs mode (see cacheAs) while expr is evaluated and puts it back afterwards, so that a whole document can be knitted from the files that are there, without a stray options() call outliving it.

Usage

withCachedResults(expr, mode = c("reuse", "offline"))

Arguments

Value

the value of expr

See Also

Other cacheAs: blessCacheAs(), cacheAsInfo()

Examples

## Not run: 
withCachedResults({
  ca <- KorAPConnection() |> collocationAnalysis("Klima", cacheAs = "klima.rds")
  freq <- KorAPConnection() |> frequencyQuery("Klima", cacheAs = "klima-freq.rds")
})

## End(Not run)