01
Where it started
My way into this question was sound. I make music, and in my MSc thesis, What are we really decoding in musical attention?, I set out to decode from EEG which instrument a listener is paying attention to. The model I built on a published method scored far higher than four channels of consumer EEG should allow, and that is what made me suspicious. Tested on songs it had never heard, it fell to chance, and a classifier given no brain data at all recovered the same labels: everything pointed to a model that recognised the songs rather than reading attention. On a dataset designed to rule that shortcut out, the models followed the statistics of the music and of their training data instead.
The thesis did not end with a better decoder. It ended with a clearer picture of what the decoders were doing, and that turned out to be the part I cared about most. I found the same pattern in other projects. In a text-to-image bias study, apparent couplings between attributes of people came from pooling prompts that each carry their own bias, not from the generator tying the attributes together. Asked plainly, GPT-4o was right about almost every single statement, yet often called both a claim and its opposite true.
Thesis code and evidence OpenInterBias results GPT-4o consistency study
02
Why it matters to me
AI systems are increasingly trusted, deployed and regulated on the strength of their evaluations. When an evaluation can be passed for the wrong reason, everything built on it inherits the mistake: the decision to deploy, the safety claim, the rule written around it. I want to help close that gap, so that what we say about a model matches what it does.
It is also a matter of values. I prefer tools I can inspect, and I publish my code together with the evidence behind each result. I would like claims about AI to meet the same standard: open to checking by anyone with a reason to doubt them.
03
How I work
I treat a good score as a hypothesis about what a model has learned, and I try to break it before I believe it: removing the shortcut I suspect, asking what a model with less information would achieve, deciding in advance what would count as success, and checking that a test can detect what it is looking for. I keep every reported number traceable to the run that produced it, and I report negative results together with what they rule out. In my thesis these habits became written method rules, each one added after something went wrong.
04
Where I am heading
The questions I want to pursue grow out of this work, and they matter most for language models and agents.
Q1
Do evaluations measure what they name?
Benchmarks and safety tests make claims about capability, robustness, privacy and harm. I want those claims to come with the controls that would expose a shortcut.
Q2
What does a model represent, and does it use it?
Interpretability methods can read a concept from a network’s activations. I want to know when that reading reflects what the model actually relies on, and to test it by intervening on the model.
Q3
Does behaviour hold where it was not tested?
Models and agents meet situations their evaluations never covered. I want evaluations that reach those situations before users do.
This is a working statement: I update it as my work and my questions change.