Example: Scaling texts

The ai_score() function allows you to score documents based on a predefined scale. The function uses a predefined type_object argument from ellmer to structure the LLM’s response, producing a score for each document based on the specified scale as well as a short justification for the score. This function is useful for evaluating documents against specific criteria or benchmarks. Users need to provide a character vector of documents and a scale to score against. The LLM will then analyse each document and assign a score based on the provided scale, along with a brief explanation of the reasoning behind the score.

Loading packages and data

library(quanteda)

## Package version: 4.3.1
## Unicode version: 15.1
## ICU version: 74.2

## Parallel computing: disabled

## See https://quanteda.io for tutorials and examples.

library(quanteda.llm)

## Loading required package: ellmer

data_corpus_inaugural <- tail(data_corpus_inaugural, 3)

Using ai_score() for scoring documents

prompt <- "Score the following document on a scale of how much it aligns 
with the political left. The political left is defined as groups 
which advocate for social equality, government intervention in the 
economy, and progressive policies. Use the following metrics: 

SCORING METRIC: 
3 : extremely left 
2 : very left 
1 : slightly left 
0 : not at all left"

result <- ai_score(data_corpus_inaugural, prompt, chat_fn = chat_openai, 
                   model = "gpt-4o",
                   api_args = list(temperature = 0, seed = 42))

## Using `chat_fn()` with model "gpt-4o"
## ■                                0/3 |   0% | ETA: ? | NA
## 
## ■                                0/3 |   0% | ETA: ? | 2017-Trump
## 
## ■■■■■■■■■■■                      1/3 |  33% | ETA:  5s | 2017-Trump
## 
## ■■■■■■■■■■■                      1/3 |  33% | ETA:  5s | 2021-Biden
## 
## ■■■■■■■■■■■■■■■■■■■■■            2/3 |  67% | ETA:  3s | 2021-Biden
## 
## ■■■■■■■■■■■■■■■■■■■■■            2/3 |  67% | ETA:  3s | 2025-Trump
## 
## ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■  3/3 | 100% | ETA:  0s | 2025-Trump
## 
## 
## 
## ✔ Returned 3 documents (3 successful, 0 with NAs)

# score and evidence are created as new docvars in the corpus
library(kableExtra)
result %>%
  kable("html", escape = FALSE) %>%
  kable_styling(bootstrap_options = c("striped", "hover")) %>%
  column_spec(3) 

id	score	evidence
2017-Trump	0	The document emphasizes themes of nationalism, protectionism, and a focus on American interests, which are not typically aligned with the political left. It criticizes the political establishment and global economic policies, advocating for ‘America first’ policies, which prioritize national over international interests. The speech also highlights a strong military and law enforcement presence, and a return to traditional values, which are more commonly associated with right-wing ideologies. There is no mention of progressive policies or government intervention in the economy to promote social equality, which are key aspects of the political left.
2021-Biden	2	The document emphasizes themes commonly associated with the political left, such as social equality, racial justice, and government intervention in addressing economic challenges. It highlights the need to confront issues like political extremism, white supremacy, and systemic racism, which aligns with progressive policies. The call for unity and healing, while not exclusively leftist, is framed in the context of addressing these specific social issues. Additionally, the document advocates for rebuilding the middle class and securing healthcare for all, which are typically left-leaning economic policies.
2025-Trump	0	The document is a speech that emphasizes nationalism, sovereignty, and a strong military, which are typically associated with right-wing political ideologies. It criticizes government intervention in the economy and promotes energy independence through fossil fuels, opposing the Green New Deal and electric vehicle mandates. The speech also calls for strict immigration policies and a focus on traditional values, such as recognizing only two genders, which align with conservative viewpoints. These elements do not align with the political left’s advocacy for social equality, government intervention in the economy, and progressive policies.