The Next Better Thing

Audio article

Checking narration… Playback speed 0.75×1×1.25×1.5×1.75×2×

Checking article narration…

There is a little dip around three kilohertz in the graph, and I have already decided that this dip might explain why a piece of music I have listened to for years has never quite surrendered its secrets to me. The singer is apparently standing behind a curtain. The strings are hiding under a bed. The drummer has been wrapped in an old blanket and placed in a cupboard.

Or perhaps I am looking at a graph at two in the morning because I have found another way to avoid deciding what to do with the rest of my life.

The graph has a rival. The rival has a smoother line, a more expensive diaphragm, and a description so lubricated with adjectives that I expect the headphones to arrive wearing a silk robe. The bass is authoritative. The treble is airy. The midrange is intimate. There is a soundstage, which is wide, and there is imaging, which is precise, and there is separation, which presumably means the instruments have been given individual rooms and are no longer allowed to breed.

And now I want to know whether I need planar magnetic drivers.

I may not need them. I still want to know.

Gear Acquisition Syndrome is the affectionate name people give to this business of researching and acquiring equipment, then researching the equipment that might correct the deficiencies of the equipment they have just acquired. Cameras, lenses, computers, amplifiers, microphones, bicycles, watches, knives, fountain pens, anything with specifications long enough to accommodate a human longing can be drawn into the procession. It is a nickname, not a medical diagnosis. Plenty of people buy things because they need them, or because they enjoy them, or because a beautifully made object is a pleasure to hold and use.

I am interested in the point where the search grows a digestive tract of its own and begins eating the activity it was supposed to feed.

A headphone driver is a rather marvelous piece of physical persuasion. In a common dynamic design, an electrical signal passes through a coil in a magnetic field, moving a diaphragm that pushes and pulls the air. A planar magnetic design distributes conductors across a thin diaphragm and uses magnetic fields to move it. Both take the electrical representation of a performance and turn it into pressure changes that meet the elaborate, wet, biological contraption on either side of my head.

That any of this works is already indecent.

Somebody sang into a microphone, the air moved a membrane, the movement became a signal, the signal was captured and processed and encoded and sent through circuitry, and now another membrane is bullying another little volume of air, and somehow I can hear a breath before a word. There are centuries of physics and manufacturing stuffed into a cup around an ear. I would like to pause over that for a moment, except the specifications have informed me that another cup might be better.

It might be. Different headphones really do sound different. Their frequency response, distortion, fit and seal can matter, as can the way they interact with the shape of a listener’s ears. A planar driver is not automatically superior to a dynamic one, any more than a horse becomes a better means of transportation merely because its anatomy is different from a bicycle’s. The design is one part of the result; execution is another.

Even a graph, that beautiful little promise of adult supervision, needs interpretation. A frequency-response curve can tell me something useful about the relative output at different frequencies under a particular measurement setup. It cannot by itself tell me exactly what I will hear, how a headphone will fit my head, whether its clamp will become annoying, or whether I will like it. Compare curves measured on different rigs without understanding how those measurements were made, and I may as well compare two portraits painted by men who were looking at different grandmothers.

In 1933, Harvey Fletcher and Wilden Munson published influential work on how the perceived loudness of a tone varies with its frequency and level. The ear is not a ruler laid politely across the spectrum. Make one presentation a little louder than another and I may mistake the extra loudness for extra detail, excitement, clarity or whatever other spiritual condiment I have been encouraged to detect. This is one reason a fair comparison requires attention to level.

I can already hear my inner reviewer protesting that he has exceptionally discerning ears. He is wearing a velvet cape made from the lining of his own colon, and I am not taking his word for it.

There is also the small problem of memory. I can recognize a song years after I last heard it. I can remember a voice, a melody, the peculiar hiss on an old recording. But holding an exact, detailed impression of timbre in memory while I remove one pair of headphones, put on another, adjust the fit, find the same passage and remember which button I was supposed to press is a different business. Auditory memory has several forms and timescales; it does not expire with a theatrical pop after a fixed number of seconds. Fine comparisons can nevertheless become difficult when there is a delay, especially when the differences are small.

Into that difficulty crawls doubt.

Was the violin clearer on the first pair? Did the second have more space between the instruments? Was I hearing the headphones, the recording, the volume, the fit, or my own expectation arriving dressed as an observation? It is astonishing how often the brain can manufacture a confident verdict while the evidence is still putting on its trousers.

The measurements are useful. The listening is useful. The uncertainty between them is also useful, especially to anyone selling me an upgrade.

A long time before headphone reviews, Denis Diderot wrote about a new dressing gown that made his other possessions look shabby. The elegant gown arrived and, in his essay, the room began to suffer by comparison. One improvement called for another improvement, and soon the original pleasure had acquired dependents. A new desk here, a better chair there, each object reproaching the others for their insufficient refinement.

My headphones do something similar to the rest of the signal chain. A better transducer leads to a question about the amplifier. The amplifier leads to a digital-to-analog converter. The converter leads to cables, files, measurements, settings and the possibility that I have spent years feeding expensive music through a microscopic corridor lined with acoustic manure.

Some of those questions have perfectly sensible answers. An amplifier can be unsuitable for a particular headphone. A bad recording can remain bad on excellent equipment, although excellent equipment may allow me to hear the badness in greater and more exquisite detail. A poor seal can change the bass I hear. The useful question is which problem exists and whether the proposed purchase actually addresses it.

The more entertaining question is why the pursuit itself can become so absorbing.

Dopamine is frequently introduced to the public as the brain’s pleasure chemical, a description so tidy that it should be sold with a complimentary bib. Dopamine is involved in several functions, including movement, motivation and learning, and the relationship between wanting something and liking it is more complicated than one chemical pressing one cheerful button. The neuroscientist Kent Berridge and others have studied the distinction between wanting and liking. A cue can acquire the power to pull us toward a reward even when the pleasure of actually receiving it is not especially grand.

In 1997, Wolfram Schultz, Peter Dayan and P. Read Montague described influential evidence connecting the activity of certain dopamine neurons to reward prediction. In simplified terms, an unexpected reward can produce a response; as an animal learns that a cue predicts the reward, the response can shift toward the cue. When an expected reward fails to appear, activity can dip. The difference between what was expected and what happened helps a learning system adjust its expectations.

One way to express a prediction error in a simple model is:

Here $r$ is the reward received, $V(s)$ is the predicted value of the current state, $V(s')$ is the predicted value of the next state, and $\gamma$ controls how much future value counts. The little $\delta$ is the difference between what the model expected and what the outcome suggests.

That is a mathematical model, not a window through which I can inspect the dopamine molecules in my head while reading headphone reviews. I cannot declare that my dopamine has peaked at the impedance graph, plunged when the package arrived and returned to its former position by the third Tuesday. Nobody has measured my neurons during this particular parade of consumer longing. People also vary, and the pleasure of a good purchase can last a long time.

Still, the distinction between anticipation and enjoyment makes uncomfortable sense to me. The search presents possibilities in a concentrated form. I can imagine the improvement without encountering its limitations. The new object has not yet had an opportunity to be merely an object.

Before purchase, it can rescue every recording. After purchase, I have to wear it.

There is another satisfaction in the hunt: it feels like work. Sometimes it is work. Researching specifications, understanding electrical loads and learning how measurements are made can teach me real things. But there is a point at which the research begins to impersonate the task it was meant to support. I can spend an afternoon comparing microphones without recording a word, or examine computing hardware with the solemnity of a man preparing to launch a spacecraft while the actual program waits, unbuilt, for a decision I could have made with the machine already available.

Buying a better instrument may make a piece of work possible. It may also offer a handsome substitute for beginning it. The substitute has a price, a delivery date and a reassuring list of features. The work has an empty page and several opportunities to discover that I am less competent than I had hoped.

Naturally, I study the specifications.

Artificial intelligence has now furnished this appetite with a new digestive system. I do not have to wait for a large cardboard box. A new model appears, a laboratory releases a table, a larger context window is announced, somebody posts a difficult programming problem and its supposedly magnificent solution, and I can begin imagining an upgrade to my thinking before I have even tried the thing.

Call it Model Acquisition Syndrome. I have no intention of issuing certificates.

The benchmark table occupies roughly the same position as the headphone graph. It offers numbers, and numbers can be genuinely informative. If one model solves 63 out of 100 test problems and another solves 60, the reported difference is three percentage points. Whether that difference means much to me depends on the questions, the test conditions, the number of examples, the possibility of contamination, the variability between runs, and whether my actual work resembles the test.

A model that improves on a mathematical benchmark may be materially better at the kind of mathematics I need. It may also be better at a narrow collection of problems that happens to be convenient for producing a triumphant chart. A coding benchmark can tell me something about coding performance; it cannot promise that the model will understand my particular codebase, preserve my design, notice an obscure failure or refrain from inventing a library function with the confidence of a hereditary prince.

Yet three percentage points are easy to see. The uncertainty around them takes longer to read.

So I conduct my own test. I give the model a proof, or an architectural problem, or a piece of prose. It handles one difficulty beautifully. There it is, the little shiver: perhaps this is the one that will finally understand what I mean without requiring me to explain it six times.

Then it makes a confident mistake in the next response. I change the prompt. It does better in one place and worse in another. I wonder whether the first failure was caused by my instructions, an unlucky sample, missing context, an inaccessible tool, the model’s actual limitations, or a mixture of all of them. The uncertainty is real. So is the temptation to keep testing until I obtain the answer I wanted.

A language model generates output by assigning probabilities to possible continuations. For a sequence of tokens, the basic idea can be written as:

The probability of a sequence depends on the probabilities assigned to each token as it follows the prompt and the tokens already produced. How a particular response is selected depends on the sampling procedure and settings. Change those, or change the instructions, tools or context, and the answer may change. Some configurations can be deterministic; other systems deliberately sample from the available possibilities.

None of this means that a probable sentence is necessarily true. A sentence can be fluent, plausible, beautifully arranged and wrong enough to send an entire afternoon down the plumbing.

This makes comparing models more slippery than comparing two pieces of hardware. The headphones can produce a repeatable response under controlled conditions, although fit, position and measurement introduce their own variations. With an AI system, even deciding what counts as the same test requires care. Was the prompt identical? Were the tools the same? Did one model get access to a file, a search result or a longer instruction that the other did not? Was a single lucky answer mistaken for reliable performance?

B. F. Skinner studied how behavior changes under different schedules of reinforcement, and it is tempting to look at a model that sometimes produces a startlingly good answer and declare that I have become a laboratory animal pressing a lever. The analogy has some bite. An uncertain result can encourage another attempt, particularly when I know that changing a prompt may genuinely improve the answer.

But an analogy is not a diagnosis, and varying responses do not prove that a company has deliberately constructed a gambling machine inside my browser. Sometimes another attempt is the sensible thing to do. The problem is that I may not know when I have moved from useful investigation into an elaborate ceremony for avoiding the result already in front of me.

The companies do have an incentive to advertise their advances, and I have an appetite for believing that the next advance will remove an irritation I have not otherwise managed to solve. Between those two interests, an astonishing quantity of ceremonial intestinal vapors can be passed off as inevitability.

There are real advances. A model that can reliably handle a task its predecessor could not is not a hallucination in my shopping cart. A larger context window can be useful, provided the system can actually use the relevant information within it. More capable local hardware can allow a model to run at an acceptable speed, or run at all. A second model can expose a mistake made by the first, although two models may share training data, assumptions and errors, and therefore cannot be treated as independent witnesses merely because their logos differ.

And sometimes the model is the bottleneck. No amount of lovingly arranged context will make an incapable system solve a problem beyond its reach. At other times the difficulty lies in the material it was given, the way the task was framed, the absence of a reliable source, a broken tool, or my own refusal to specify what a successful result would look like.

The irritating part is that improvement does not announce which part of the system needs improving.

I can buy the most celebrated headphones and feed them a poor recording. I can use the most celebrated model and give it a vague problem, irrelevant documents and no way to check its answer. In either case the new apparatus may reveal something, but it cannot be expected to repair every defect upstream while also flattering me for having purchased it.

A better comparison need not be a grand scientific undertaking. For headphones, I can use familiar recordings, match levels as carefully as practical, switch without knowing which pair I am hearing when possible, and repeat the comparison. For models, I can collect a small set of tasks that resemble the work I actually do, decide what a good answer should contain, use comparable instructions and tools, and check the results rather than rewarding whichever answer sounds most assured.

If the supposedly superior object makes a clear difference, splendid. If the difference is small, inconsistent or irrelevant to my purpose, that is information too. I am still allowed to prefer the more expensive one. Pleasure does not need to submit a budget proposal, although the budget may have a few objections.

Richard Feynman warned that the first principle is that “you must not fool yourself,” adding that “you are the easiest person to fool.” I find this offensive because it leaves me very little room to blame the headphones.

The trouble is that curiosity and acquisition are tangled together. I do want to know how a diaphragm moves, how a graph is measured, why two models answer differently, what a larger model can do that a smaller one cannot. These questions are not empty. The machinery is extraordinary. The people who worked out parts of it were not all laboring for my eventual convenience, yet here I am with their discoveries assembled into products and placed before me along with monthly payment options.

I would like to understand them without being dragged by them. I would also like the next model to be better, the next recording to disclose something I missed, and the next graph to settle the question that the previous six graphs have somehow made more urgent.

I still want to know what that dip around three kilohertz sounds like.

P.S. References

Wolfram Schultz, Peter Dayan and P. Read Montague, “A Neural Substrate of Prediction and Reward,” 1997: https://doi.org/10.1126/science.275.5306.1593

Kent C. Berridge, “The Debate Over Dopamine’s Role in Reward: The Case for Incentive Salience,” 2007: https://doi.org/10.1007/s00213-006-0578-x

Harvey Fletcher and Wilden A. Munson, “Loudness, Its Definition, Measurement and Calculation,” 1933: https://doi.org/10.1002/j.1538-7305.1933.tb00403.x

Percy Liang and colleagues, “Holistic Evaluation of Language Models,” 2022: https://arxiv.org/abs/2211.09110

Richard Feynman, “Cargo Cult Science,” 1974: https://calteches.library.caltech.edu/51/2/CargoCult.htm