Reasoning models don't so much think as navigate

The new lan­guage models are chil­dren of the rea­soning rev­o­lu­tion, and they stream out these long, cir­cuitous thinking traces. They are said to be applying more com­pute to our ques­tions and challenges.

This is subtle, but that “more” isn’t par­tic­u­larly about thinking harder. Rather, it’s about thinking in the right direc­tion. It is not the gas pedal, but the steering wheel — better yet, the GPS map in the dashboard.

The rea­soning rev­o­lu­tion depends, in part, on the unrea­son­able effec­tive­ness of spe­cific words: twists like “but wait” and “actually”, which operate as pow­er­fully as magic spells. (The Eng­lish depart­ment NEEDS to get into the game with this stuff.) Is the phrase “but wait” really a white-hot kernel of intel­lec­tual effort? No. It’s a sign planted in the ground, pointing THAT-A-WAY, towards a par­tic­ular kind of doc­u­ment that humans find useful.

(Don’t mis­take pre­ci­sion for minimization. I’m not dis­mis­sively saying, these are just doc­u­ments; I am plainly observing, these are doc­u­ments. If you don’t think doc­u­ments are cool, even some­times cosmic, that’s on you!)

Notice that, as in real life, direc­tions aren’t always cor­rect. It’s likely that you have by now watched a lan­guage model walk in circles, “but wait”-ing itself back around, and around, and around again … 

Recent research from Apple talks about “forks” in the road, with “distractors” that can lead a model in the wrong direc­tion.

Here’s more evi­dence for the nav­i­ga­tion argument: base models can already do the things rea­soning models can do … it just takes them much longer to arrive in the cor­rect regions of high-dimensional space. Base models are fine thinkers, but cruddy navigators.

The single for­ward pass of a lan­guage model runs on its own, refracting a con­text window into an array of probabilities; that’s all “the model” ever does. However, each for­ward pass can “stand on the shoul­ders of giants”, taking direc­tion from pre­vious passes, bringing its brief labors into better align­ment with the desires of the human operator, way out there.

As usual, obser­va­tions about lan­guage models raise ques­tions about human minds. Do we think harder mostly by thinking in the right direc­tion? I think the answer is some­times yes — thinking as search — and some­times no. Maybe I’m wrong, but I believe I can feel dif­ferent mech­a­nisms at work. And of course human thought is not a doc­u­ment; it unfurls, and compounds, and con­siders itself, in a richer space.

(This post is related to the latest edition of my pop-up newsletter on AI.)