Applying statistics to LLM evaluations

Overview

There is a need to evaluate the performance and quality of LLM-based applications at the ABS. Such evaluations can be complex, and often depend not only on the LLM, but on its surrounding architecture, the use-case task it is performing, and the business domain.  The ABS are developing their capability in designing and running these evaluations ("evals" ) for LLM-based use-cases. To date, running "evals" typically involves testing a model’s performance over a finite evaluation dataset, with no consideration for whether these results are statistically significant or not. To maximize informativeness and minimize uncertainty it is important to go beyond reporting single numbers for evaluation scores, "e.g. the evaluation metric is 80% correct on our evaluation dataset", to providing more robust statistical estimates, including measures of variance in the estimator, and robust confidence intervals.

Research engagement

For this project, the student would conduct a literature review, expanding on the overview Applying Statistics to LLM Evaluations and then develop scripts to implement these statistical methods, in R or python, for testing against current ABS evaluation pipelines and scores.

Research activities

Students will engage in the literature review and software development with A/Prof Gentry White and ABS personnel.

Research skills

For this project, the student would conduct a literature review, expanding on the overview Applying Statistics to LLM Evaluations and then develop scripts to implement these statistical methods, in R or Python, for testing against current ABS evaluation pipelines and scores.

Outcomes

For this project, the student would conduct a literature review, expanding on the overview Applying Statistics to LLM Evaluations and then develop scripts to implement these statistical methods, in R or Python, for testing against current ABS evaluation pipelines and scores.

Skills and experience

Statistics or Applied Mathematics or Computer Science

Start date

2 November, 2026

End date

19 February, 2027

Location

GP Y Block Level 8

Keywords

• Large Language Models
• Model Evaluation
• Validation

Contact

Gentry White

3138 1658

whiteg5@qut.edu.au