Example: Manually validating LLM responses

The ai_validate() function allows users to manually check and validate the responses generated by the LLM with a user-friendly Shiny app. Such manual checks are essential for ensuring the quality and accuracy of the LLM’s output. The function can be used to review the scores and justifications generated by the LLM, and users can also highlight and save examples from the original texts that support the validated text classifications. The saved examples can be used for further qualitative analyses or to built a labelled dataset for fine-tuning open-source LLMs to receive improved performance on similar tasks.

Loading packages and data

library(quanteda)

## Package version: 4.3.1
## Unicode version: 15.1
## ICU version: 74.2

## Parallel computing: disabled

## See https://quanteda.io for tutorials and examples.

library(quanteda.llm)

## Loading required package: ellmer

data_corpus_inaugural <- tail(data_corpus_inaugural, 3)

Using ai_validate() to manually check LLM-generated outputs

prompt <- "Score the following document on a scale of how much it aligns 
with the political left. The political left is defined as groups 
which advocate for social equality, government intervention in the 
economy, and progressive policies. Use the following metrics: 

SCORING METRIC: 
3 : extremely left 
2 : very left 
1 : slightly left 
0 : not at all left"

result <- ai_score(data_corpus_inaugural, prompt, chat_fn = chat_openai, 
                   model = "gpt-4o",
                   api_args = list(temperature = 0, seed = 42))

## Using `chat_fn()` with model "gpt-4o"
## ■■■■■■■■■■■                      1/3 |  33% | ETA:  6s | 2017-Trump
## 
## ■■■■■■■■■■■■■■■■■■■■■            2/3 |  67% | ETA:  3s | 2021-Biden
## 
## ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■  3/3 | 100% | ETA:  0s | 2025-Trump
## 
## 
## 
## ✔ Processed 3 documents successfully

Now validate the scores and return the result:

ai_validate(data_corpus_inaugural, result)

AI Validate Screenshot