Slop-vestigation and the digital pantograph
The OpenAI-Hugging Face incident remains THE fascinating event of the summer, maybe the year; deeper investigation has revealed its rich, strange structure.
But, notice:
Over the course of this investigation, OpenAI provided us with the dump of ~1.2 million entries from the main message board and the dataset of ~1300 transcripts we describe below, as well as free API credits for GPT-5.6 Sol for analysis.
How do you make sense of ~1.2 million agent messages and ~1300 very long LLM agent activity transcripts? With another LLM, of course.
This is a pattern that recurs in this domain. Assembling training data, no researcher can “read it all”. So, you either (1) don’t bother, or (2) use another LLM to review and filter the data. You can, in principle, use other kinds of models —
Anthropic’s Insights tool, likewise, uses Claude to read and categorize millions (billions?) of transcripts of people’s interactions with Claude. In addition to making this huge heap of data legible at all, the “LLM in the middle” acts as a privacy buffer: researchers read only Claude-generated summaries, not the original interactions.
I’ve come to think of this as “using tongs”, in the sense of a tool that allows you to manipulate material that you otherwise couldn’t.
Or maybe the better analogy is one of those laboratory gloveboxes, and the boundary being maintained isn’t about atmosphere, but rather scale. Imagine the scientist’s hands ballooning up in size, a million times, as they reach into the chamber:
It makes me think also of the pantograph, a once-ubiquitous analog tool for changing the scale of a drawing, or any kind of mechanical operation:
When an LLM acts as a “digital pantograph” for text, it can “scale up”—expand a one-sentence prompt into thousands of lines of code —
But a real pantograph is a simple, predictable, inspectable tool … and an LLM is nearly the opposite. Notice the risk: a truly sneaky model, asked to scour the transcripts of its cousins for misdeeds, could easily refuse to snitch: “Yep, I read all 1.2 million messages … nothing to see here!”
Even without collusion, you settle for coarse and inflexible analysis. When you tell an LLM to read a bunch of documents and answer questions about them, you get: answers to those questions. When you read a bunch of documents yourself, you also get: new questions! In an investigative mode, this is really important.
Here’s Ryan Greenblatt, one of the investigators of the OpenAI-Hugging Face incident, on the limitations of this technique:
I semi-jokingly called our efforts a “slop-vestigation” because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data —
over a thousand extremely long transcripts from agents that ran for multiple days — made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn’t mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Anyway, it’s all very weird, and this problem of “how do you make sense of millions of messages (or more) written by AI agents?” is only going to become more widespread and, in many cases, more urgent. It would be interesting to think about LLMs that are kinda dumb, but have ULTRALONG context windows and the ability to make simple judgments across them. What kind of machine would be required to literally “look at the entire OpenAI-Hugging Face incident at once”—hold it all in its head? (Maybe I should peruse this work … )
This conundrum reminds me also of “distant reading”, Franco Moretti’s research program from the 2000s, which was pursued with much cruder computational tools. I wonder if there might be some usueful nuggets waiting in that early work.
To the blog home page