The ai_score() function allows you to score documents
based on a predefined scale. The function uses a predefined
type_object argument from ellmer to structure
the LLM’s response, producing a score for each document based on the
specified scale as well as a short justification for the score. This
function is useful for evaluating documents against specific criteria or
benchmarks. Users need to provide a character vector of documents and a
scale to score against. The LLM will then analyse each document and
assign a score based on the provided scale, along with a brief
explanation of the reasoning behind the score.
Loading packages and data
## Package version: 4.3.1
## Unicode version: 15.1
## ICU version: 74.2## Parallel computing: disabled## See https://quanteda.io for tutorials and examples.## Loading required package: ellmer
data_corpus_inaugural <- tail(data_corpus_inaugural, 3)Using ai_score() for scoring documents
prompt <- "Score the following document on a scale of how much it aligns
with the political left. The political left is defined as groups
which advocate for social equality, government intervention in the
economy, and progressive policies. Use the following metrics:
SCORING METRIC:
3 : extremely left
2 : very left
1 : slightly left
0 : not at all left"
result <- ai_score(data_corpus_inaugural, prompt, chat_fn = chat_openai,
model = "gpt-4o",
api_args = list(temperature = 0, seed = 42))## Using `chat_fn()` with model "gpt-4o"
## ■■■■■■■■■■■ 1/3 | 33% | ETA: 7s | 2017-Trump
##
## ■■■■■■■■■■■■■■■■■■■■■ 2/3 | 67% | ETA: 4s | 2021-Biden
##
## ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 3/3 | 100% | ETA: 0s | 2025-Trump
##
##
##
## ✔ Processed 3 documents successfully
# score and evidence are created as new docvars in the corpus
library(kableExtra)
result %>%
kable("html", escape = FALSE) %>%
kable_styling(bootstrap_options = c("striped", "hover")) %>%
column_spec(3) | id | score | evidence |
|---|---|---|
| 2017-Trump | 0 | The document emphasizes themes of nationalism, protectionism, and a focus on American interests, which are not typically aligned with the political left. It criticizes the political establishment and globalism, advocating for policies like ‘America first,’ which prioritize national over international concerns. The speech also highlights reducing government intervention in favor of empowering citizens directly, which contrasts with the left’s preference for government-led solutions to social and economic issues. Overall, the document does not align with the political left’s advocacy for social equality and progressive policies. |
| 2021-Biden | 2 | The document emphasizes themes of unity, social justice, and government intervention, which align with the political left. It highlights the need to address racial justice, climate change, and economic inequality, advocating for policies that promote social equality and government action. The call for unity and healing, while not exclusively leftist, is framed in the context of addressing systemic issues and promoting progressive change, which are key aspects of left-leaning ideology. |
| 2025-Trump | 0 | The document is a speech that emphasizes nationalism, sovereignty, and a strong military, which are typically associated with right-wing ideologies. It criticizes government intervention in the economy, such as the Green New Deal and electric vehicle mandates, and promotes energy independence through drilling, which aligns with conservative economic policies. The speech also focuses on border security and immigration control, which are key issues for the political right. Additionally, it calls for a colorblind and merit-based society, opposing social engineering based on race and gender, which contrasts with progressive policies that advocate for social equality and diversity. |