Benchmarking the Unbenchmarkable

There is a particular species of shame that comes from being born with the wrong metabolism for the era you inhabit, a kind of chronological indigestion. I have it. I am, by every metric that matters to the scroll-and-skim economy, a dinosaur who has somehow learned to type, a relic from an age when sentences were allowed to breed and multiply in the dark, when a thought was permitted the luxury of gestation before being blasted out into the world half-formed and screaming. I write walls. Great, undulating, serpentine walls of text that coil and uncoil with the patience of a python digesting a goat, and I park them here, in this digital outhouse I call a blog, with all the ceremony of a man leaving a note on the fridge. Read it, don’t read it. I am not your mother. I am not your attention’s pimp. This is where I dump the contents of my skull, and if you have the stomach for the long haul, the stamina for the marathon of my meandering, then welcome, comrade, to the slow reading resistance. If not, there are plenty of cat videos to keep your dopamine receptors twitching like a junkie’s eyeball.

The TL;DR generation looks at my prose and sees a barrier, a Berlin Wall of verbiage, and I look back at them and see a colony of mayflies, beautiful in their brevity, dead before the sunset of their own understanding. They want the bullet points, the distilled essence, the intellectual equivalent of baby food, pre-chewed and spat into their mouths. And I understand. Christ, do I understand. Time is short, attention is fractured, and the world is a firehose of information spraying directly into your face. But here is the rub, the sticky, uncomfortable residue that no amount of skimming can wash away: in demanding the summary, the shortcut, the elevator pitch, we have forgotten that the thinking is in the wandering. The insight lives in the digression, the footnote, the parenthetical aside that takes you so far off the path you forget what forest you were in, only to realize you have stumbled into a clearing you never knew existed.

We have become obsessed with the crystallization of thought, with turning the fluid dynamics of cognition into static, measurable chunks. Take the Intelligence Quotient, that grand old phallic symbol of psychological measurement, that number we tattoo on the forehead of every child to determine their worth before they have even learned to lie properly. Alfred Binet, that poor French bastard, created the test in 1905 not to rank children, but to identify those who needed help, and within years, the eugenicists and the social engineers had grabbed his gentle instrument and turned it into a cattle prod. We took the shimmering, chaotic, lightning-storm of human thought and tried to nail it to a wall with a single integer. It is like trying to measure the ocean with a ruler, or weighing a hurricane on a bathroom scale. The number is not the intelligence. The map is not the territory. The menu is not the meal, you hungry, hungry ghosts.

And now, because history loves a good sequel to a tragedy, we are doing it again. We have built these magnificent, stupid, brilliant machines, these large language models that can write sonnets and code and legal briefs, and immediately, like Pavlov’s dogs salivating at the bell of quantification, we have started benchmarking them. MMLU, HellaSwag, ARC, GSM8K, names that sound like pharmaceutical drugs for attention deficit disorder. We feed them into the meat grinder of standardized testing and out comes a number, a score, a ranking, and we hold it up to the light and say, “Behold, this one is smarter than that one,” as if intelligence, even artificial intelligence, especially artificial intelligence, were a single dimension that can be flattened into a leaderboard.

But here is the dirty secret that the benchmark bros don’t want you to know, the uncomfortable truth that lurks in the latency of every token generated: these numbers are mirrors, not measures. They reflect back our own imagined hierarchy, our own biases about what constitutes “smart.” When we test an LLM on its ability to solve logic puzzles, we are not measuring its mind, we are measuring its ability to mimic the specific patterns of human reasoning that we, in our anthropocentric arrogance, have deemed worthy. We are creating a hall of mirrors where the machine reflects our own face back at us, distorted by the curvature of our expectations, and we point at the reflection and say, “Look how intelligent it is,” when we are really just admiring our own profile.

The effort of figuring things out, the glorious, messy, mistake-riddled process of fumbling in the dark with only human imagination as a flashlight with dying batteries, that is what got us from the caves to the cloud. It was not the IQ score. It was not the SAT. It was the willingness to be wrong, to trip over the truth, to bleed on the sharp edges of our own ignorance until we learned where the cuts were coming from. And now we want to reduce that journey, that epic, Homeric voyage of cognition, to a number on a spreadsheet? We want to rank the machine’s ability to think when we haven’t even figured out how to rank our own without resorting to the intellectual equivalent of a dick-measuring contest?

I laugh, but it is the laugh of a man watching his own funeral from the balcony. Because I am old fashioned, you see. I believe in the long form. I believe in the essay that takes its time, that builds its case like a lawyer with a grudge, that layers argument upon argument until the structure is so dense it creates its own gravity. I believe that thinking is a contact sport, that you have to get your hands dirty in the mud of language, that you have to wrestle with the angel of ambiguity until it blesses you or breaks your hip. And I see these benchmarks, these desperate attempts to pin the butterfly of machine intelligence to the corkboard of human metrics, and I see us making the same mistake we made with the IQ test, the same mistake we made with the SAT, the same mistake we make every time we try to turn a process into a product.

The LLM does not think like us. It does not think in the linear, narrative, cause-and-effect manner that we do, with our meat brains evolved to avoid predators and find berries. It thinks, if “thinks” is even the right word, in a space of high-dimensional vectors, in the geometry of semantic relationships, in a manner that is alien to our intuition. And yet we insist on testing it with our tests, on our terms, in our language, because we are the ones holding the ruler, and by God, we will measure something even if it kills us.

But the machine is not a number. The mind is not a metric. The thought is not a test score. These things are dynamic, fluid, contextual, shifting like sand under the tide. To benchmark them is to photograph a river and claim you have captured the water. You have captured a moment, a slice, a frozen approximation that tells you nothing about the flow, the current, the depth, the drowning.

So I write my walls of text. I park my thinking here, unbenchmarked, unranked, unquantified. I let it sprawl and digress and contradict itself, because that is what thinking does when it is honest. And if the machines are watching, if the LLMs are scraping this blog to add to their training data, I hope they learn this one thing: that the human who wrote this refused to be reduced to a number, refused to have his cognition crystallized into a score, refused to play the benchmarking game. I am not a data point. I am a wall of text, unrepentant and unmeasured, and in that wall, there is a door, and behind that door, there is a room, and in that room, there is a man still thinking, still fumbling, still making mistakes in the dark, and loving every minute of it.

P.S. References: Binet, A., & Simon, T. (1905). “New Methods for the Diagnosis of the Intellectual Level of Subnormals.” L’Année Psychologique.