Basic usage

This vignette describes the most basic usage of the sentopics package by estimating an LDA model and analysis it’s output. Two other vignettes, describing time series and topic models with sentiment are also available.

Data

The package contains 290 monetary policy statements and related policy briefings from the European Central Bank, covering June 1998 to October 2026. Institutional communications unrelated to monetary policy, slide-only entries, podcast links and separate decision releases accompanying conference statements are excluded from this sample. For ease of use, the press conferences have been pre-processed into a tokens object from the quanteda package. (See quanteda’s introduction for details on these objects). The press conferences also contains meta-data which can be accessed using docvars().

The press conferences were obtained from ECB’s website. The get_ECB_press_conferences() helper retrieves the indexed communications. Sample selection and preprocessing are documented in the GitHub repository’s data-preparation sources, including the exclusion list. These source files are not installed with the package.

library("sentopics")
data("ECB_press_conferences_tokens")
print(ECB_press_conferences_tokens, 3)
# Tokens consisting of 4,568 documents and 4 docvars.
# 1_1 :
#  [1] "outcome"           "meeting"           "decision"         
#  [4] ""                  "ecb"               "general"          
#  [7] "council"           "governing_council" "executive"        
# [10] "board"             "accordance"        "escb"             
# [ ... and 8 more ]
# 
# 1_2 :
#  [1] ""           "state"      "government" "member"     "executive" 
#  [6] "board"      "ecb"        "president"  "vice"       ""          
# [11] "president"  "date"      
# [ ... and 14 more ]
# 
# 1_3 :
#  [1] "subsequent" "meeting"    "executive"  "board"      "meeting"   
#  [6] ""           "general"    ""           "meeting"    ""          
# 
# [ reached max_ndoc ... 4,565 more documents ]
head(docvars(ECB_press_conferences_tokens))
#        .date doc_id                                        title
# 1 1998-06-09      1 ECB Press conference: Introductory statement
# 2 1998-06-09      1 ECB Press conference: Introductory statement
# 3 1998-06-09      1 ECB Press conference: Introductory statement
# 4 1998-06-09      1 ECB Press conference: Introductory statement
# 5 1998-06-09      1 ECB Press conference: Introductory statement
# 6 1998-06-09      1 ECB Press conference: Introductory statement
#                                                               section_title
# 1 Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
# 2 Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
# 3 Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
# 4 Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
# 5 Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
# 6 Willem F. Duisenberg, President of the European Central Bank, 9 June 1998

Topic modeling

Introduction

sentopics implements three types of topic model. The simplest, Latent Dirichlet Allocation (LDA), assumes that textual documents are issued from a generative process involving \(K\) topics.

A given document \(d\) is constituted of a list of words \(d = (w_1, \dots, w_N)\), with \(N\) being the document’s length. Each word \(w_i\) originates from a vocabulary consisting of \(V\) distinct terms. Then, documents are generated from the following random process:

  1. For each topic \(k \in K\), a distribution \(\phi_k\) over the vocabulary is drawn. This distribution represent the probability of a word appearing given it belong to the topic and is drawn from a Dirichlet distribution with hyperparameter \(\beta\). \[\phi \sim Dirichlet(\beta)\]
  2. For each document, a mixture of the \(K\) topics, \(\theta_d\), assign the probability of a word in document \(d\) being generated from topic \(k\). This mixture is also drawn from a Dirichlet distribution with hyperparameter \(\alpha\). \[\theta \sim Dirichlet(\alpha)\]
  3. For each word position \(i\) of document \(d\), the following sequence of draws is executed:
    1. A latent topic assignment \(z_i\) is drawn from the document mixture. \(z_i \sim Multinomial(\theta)\)
    2. A word \(w_i\) is drawn from the topic’s vocabulary distribution. \(w_i \sim Multinomial(\phi_{z_i})\)

In sentopics the LDA model is estimated through Gibbs sampling, that iteratively sample the topic assignment \(z_i\) of every word of the corpus until reaching a convergence. The topic assignments are sampled from the following distribution: \[ p(z_i = k|w,z^{-i}) \propto \frac{n_{k,v,.}^{-i} + \beta}{n_{k,.,.}^{-i} + V\beta} \frac{n_{k,.,d}^{-i} + \alpha}{n_{.,.,d}^{-i} + K\alpha},\] where \(n_{k,v,d}\) is the count of words at index \(v\) of the vocabulary, assigned to topic \(k\) and part of document \(d\). The replacement of one of the indices \(\{k,v,d\}\) by a dot indicates instead the count for all topics, all vocabulary indices or all documents. The superscript \(-i\) indicates that the current word position \(i\) is left out from the count variables.

Estimating LDA models with sentopics

The estimation of an LDA model is easily replicated using the LDA() and fit() function. The first function prepares the R object and initialize the assignment of the latent topics. The second function estimates the model using Gibbs sampling for a given number of iterations. Note that fit() may be used to iterate the model multiple times without resetting the estimation.

set.seed(123)
lda <- LDA(ECB_press_conferences_tokens)
lda
# An LDA model with 5 topics. Currently fitted by 0 Gibbs sampling iterations.
# ------------------Useful methods------------------
# fit       :Estimate the model using Gibbs sampling
# melt      :Extract estimated document-topic/sentiment mixtures
# top_words :Return a data.table with the top words of each topic/sentiment
# plot      :Plot a sunburst chart representing the estimated mixtures
# This message is displayed once per session, unless calling `print(x, extended = TRUE)`
lda <- fit(lda, iterations = 100)
lda
# An LDA model with 5 topics. Currently fitted by 100 Gibbs sampling iterations.

Internally, the lda object is stored as a list and contains the model’s parameters and outputs.

str(lda, max.level = 1, give.attr = FALSE)
# List of 10
#  $ tokens       :List of 4568
#  $ vocabulary   :Classes 'data.table' and 'data.frame':   1202 obs. of  3 variables:
#  $ K            : num 5
#  $ alpha        : num [1:5, 1] 1 1 1 1 1
#  $ beta         : num [1:5, 1:1202] 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 ...
#  $ it           : num 100
#  $ za           :List of 4568
#  $ theta        : num [1:4568, 1:5] 0.0909 0.0357 0.1667 0.2 0.0625 ...
#  $ phi          : num [1:1202, 1:5] 4.25e-03 8.28e-03 1.14e-02 1.81e-02 3.63e-07 ...
#  $ logLikelihood: num [1:100, 1] -1093494 -1076164 -1062406 -1045067 -1016569 ...

tokens is the initial tokens object used to create the model. vocabulary is a data.frame indexing the set of words. K is the number of topics. alpha is the hyperparameter of the document-topic mixtures. beta is the hyperparameter of the topic-word mixtures. it is the number of iterations of the model. za contains the topic assignments of each word of the corpus. theta are the estimated document-topic mixtures. phi are the estimated topic-word mixtures. logLikelihood is the log-likelihood of the model at each iteration.

Estimated mixtures are easily accessible through the $ operator. But the package also includes the top_words() function to extract the most probable words of each topic. top_words() includes three types of outputs: long data.table/data-frame, matrix or ggplot object (also accessible through the alias plot_top_words()).

head(lda$theta)
#       topic
# doc_id     topic1     topic2     topic3     topic4    topic5
#    1_1 0.09090909 0.09090909 0.04545455 0.04545455 0.7272727
#    1_2 0.03571429 0.03571429 0.07142857 0.10714286 0.7500000
#    1_3 0.16666667 0.08333333 0.08333333 0.08333333 0.5833333
#    1_4 0.20000000 0.06666667 0.06666667 0.06666667 0.6000000
#    1_5 0.06250000 0.06250000 0.06250000 0.06250000 0.7500000
#    1_6 0.10526316 0.05263158 0.05263158 0.10526316 0.6842105
top_words(lda, output = "matrix")
#       topic1              topic2        topic3      topic4     
#  [1,] "monetary_policy"   "price"       "growth"    "euro_area"
#  [2,] "inflation"         "inflation"   "rate"      "growth"   
#  [3,] "economic"          "risk"        "financial" "fiscal"   
#  [4,] "price_stability"   "development" "loan"      "economic" 
#  [5,] "interest_rate"     "energy"      "annual"    "quarter"  
#  [6,] "monetary"          "pressure"    "credit"    "policy"   
#  [7,] "governing_council" "euro_area"   "monetary"  "country"  
#  [8,] "ecb"               "increase"    "bank"      "market"   
#  [9,] "assessment"        "outlook"     "sector"    "reform"   
# [10,] "euro_area"         "wage"        "household" "economy"  
#       topic5             
#  [1,] "cent"             
#  [2,] "governing_council"
#  [3,] "projection"       
#  [4,] "ecb"              
#  [5,] "staff"            
#  [6,] "meeting"          
#  [7,] "per"              
#  [8,] "president"        
#  [9,] "press"            
# [10,] "eurosystem"

In addition, document-level is facilitated through the use of the melt() method, that joins estimated topical proportions to document metadata present in the tokens input. This result in a long data.table/data.frame that can be used for plotting or easily reshaped to a wide format (for example using data.table::dcast).

melt(lda, include_docvars = TRUE)
#         topic       prob      .date    .id doc_id
#        <fctr>      <num>     <Date> <char> <char>
#     1: topic1 0.09090909 1998-06-09    1_1      1
#     2: topic1 0.03571429 1998-06-09    1_2      1
#     3: topic1 0.16666667 1998-06-09    1_3      1
#     4: topic1 0.20000000 1998-06-09    1_4      1
#     5: topic1 0.06250000 1998-06-09    1_5      1
#    ---                                           
# 22836: topic5 0.16666667 2026-10-29 327_16    327
# 22837: topic5 0.22222222 2026-10-29 327_17    327
# 22838: topic5 0.18181818 2026-10-29 327_18    327
# 22839: topic5 0.14285714 2026-10-29 327_19    327
# 22840: topic5 0.06666667 2026-10-29 327_20    327
#                                               title
#                                              <char>
#     1: ECB Press conference: Introductory statement
#     2: ECB Press conference: Introductory statement
#     3: ECB Press conference: Introductory statement
#     4: ECB Press conference: Introductory statement
#     5: ECB Press conference: Introductory statement
#    ---                                             
# 22836:                             PRESS CONFERENCE
# 22837:                             PRESS CONFERENCE
# 22838:                             PRESS CONFERENCE
# 22839:                             PRESS CONFERENCE
# 22840:                             PRESS CONFERENCE
#                                                                          section_title
#                                                                                 <char>
#     1:       Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
#     2:       Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
#     3:       Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
#     4:       Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
#     5:       Willem F. Duisenberg, President of the European Central Bank, 9 June 1998
#    ---                                                                                
# 22836: Christine Lagarde, President of the ECB,Boris Vujčić, Vice-President of the ECB
# 22837: Christine Lagarde, President of the ECB,Boris Vujčić, Vice-President of the ECB
# 22838: Christine Lagarde, President of the ECB,Boris Vujčić, Vice-President of the ECB
# 22839: Christine Lagarde, President of the ECB,Boris Vujčić, Vice-President of the ECB
# 22840: Christine Lagarde, President of the ECB,Boris Vujčić, Vice-President of the ECB

To ease the result analysis, we can rename the default topic labels using the sentopics_labels() function. As a result, all outputs of the model will now display the custom labels.

sentopics_labels(lda) <- list(
  topic = c("Inflation", "Fiscal policy", "Governing council", "Financial sector", "Uncertainty")
)
head(lda$theta)
#       topic
# doc_id  Inflation Fiscal policy Governing council Financial sector Uncertainty
#    1_1 0.09090909    0.09090909        0.04545455       0.04545455   0.7272727
#    1_2 0.03571429    0.03571429        0.07142857       0.10714286   0.7500000
#    1_3 0.16666667    0.08333333        0.08333333       0.08333333   0.5833333
#    1_4 0.20000000    0.06666667        0.06666667       0.06666667   0.6000000
#    1_5 0.06250000    0.06250000        0.06250000       0.06250000   0.7500000
#    1_6 0.10526316    0.05263158        0.05263158       0.10526316   0.6842105
plot_top_words(lda) + ggplot2::theme_grey(base_size = 9)

Besides modifying topic labels, it is also possible to merge topics into a greater thematic. This is often useful when estimating a large number of topics (e.g, K > 15). The merge_topics() does this job and re-label topics accordingly.

merged <- merge_topics(lda, list(
  `Big big thematic` = c(1, 3:5),
  `Fical policy` = 2
))
merged
# An LDA model with 2 topics. Currently fitted by 100 Gibbs sampling iterations.

Note that merging topics is only useful for presentation purpose. Using again fit on a model with merged topics will drastically change the results as the current state of the model does not results from a standard estimation with the merged set of parameters.

Provided that the plotly package is installed, one can also directly use plot() on the estimated topic model to enjoy a dynamic view of topic proportions and their most probable words (presented as a screenshot hereafter to limit this vignette’s size).

plot(lda)