Robin Sloan
the lab
July 2021

Ghost faves in the mystery machine

Hamlet's friend restrain him while a ghost beckons; a mono­chrome etching, fine-lined, almost like a comic book illustration.

Hamlet, Horatio, Marcellus and the Ghost, 1796, Robert Thew

Recently, I was chat­ting with a friend about a prob­lem they’d encoun­tered after run­ning my script to delete their old Twit­ter faves. (These are tech­ni­cally “likes” now, I know, but I will stub­bornly call them faves, because it is the superior word.)

The prob­lem was that, although the API reported a suc­cess­ful unfaving, many old faves were still attached to my friend’s profile. Here’s the strange part: when viewed, the tweets them­selves showed hol­low hearts, vis­i­bly unfaved … yet there they remained, a ghostly list, some­how both fave and not.

Even stranger — at this point it’s get­ting delicious — the ghost faves could be ban­ished at last by refaving and then unfaving them; by power-cycling the little heart.

Cursory inves­ti­ga­tion indi­cates this is a wide­spread prob­lem. A search for “twitter phantom likes” will reveal many peo­ple describing the same behav­ior exactly, with no evi­dence of a solu­tion anywhere. Twit­ter even released a fresh new API endpoint for man­ag­ing faves — and still, the ghosts are beyond its reach.

This is clearly a bug — the API is not doing what it’s sup­posed to do, or even what it “thinks it’s doing”—but I am not here to dis­cuss a bug; rather, what the bug made me think about.

My under­stand­ing of large internet sys­tems was trans­formed when I read about Face­book’s Mys­tery Machine. This was a tool doc­u­mented back in 2014; I assume it is not in use anymore, but/and, to me, it’s still exem­plary of the way these sys­tems work, or don’t.

Face­book’s paper lays out the prob­lem:

Current debug­ging and opti­miza­tion meth­ods scale poorly to deal with the com­plex­ity of mod­ern Inter­net services, in which a sin­gle request trig­gers par­al­lel exe­cu­tion of numer­ous het­ero­ge­neous soft­ware com­po­nents over a dis­trib­uted set of computers.

In other words: how do you ana­lyze or debug a system made up of many different com­po­nents, writ­ten by many different peo­ple in many different pro­gram­ming languages, run­ning in many different places at many dif­fer­ent times … that activate each other in com­pli­cated cascades? A sys­tem that grew organically, and very quickly? That was perhaps, heh, not per­fectly doc­u­mented along the way?

A “rational” answer might be: you write some code that cap­tures an X-ray of what’s happening, and you put that code into all those com­po­nents! I mean, this kind of soft­ware exists. But the real­ity of Face­book circa 2014, its messi­ness and scale, meant that wasn’t going to happen.

So what did they do instead?

Consequently, we develop a tech­nique that gen­er­ates a causal model of sys­tem behavior with­out the need to add sub­stan­tial new instru­men­ta­tion or man­u­ally generate a schema of appli­ca­tion behav­ior. Instead, we gen­er­ate the model via large-scale rea­son­ing over indi­vid­ual soft­ware component logs.

In plain language: they watched the whole sys­tem as it ran, the parts of it they could see — the log messages — and inferred its operation from that activity, the way it played out in time. The Mys­tery Machine worked by lit­er­ally hypothesizing “maybe X causes Y … ” for a few mil­lion dif­fer­ent Xs and Ys, test­ing each hypoth­e­sis against the logs until enough had been “disproven” that a clear-ish pic­ture remained. It sounds to me more like botany or zool­ogy than engi­neer­ing or architecture!

I find this totally cosmic: Facebook in 2014, by then already a VERY large and rich company — a pow­er­ful force in the world — did not under­stand, in a pretty deep way, how Face­book worked.

Maybe Facebook at the height of its “move fast and break things” era rep­re­sents an extreme case. But the prob­lem described in the Mys­tery Machine paper is not uncommon; the heterogenous, asyn­chro­nous nature of large inter­net sys­tems seems to pro­duce it almost inevitably. An “application” like Face­book or Fort­nite is just VERY dif­fer­ent from an appli­ca­tion like Pho­to­shop or Street Fighter II.

So, at this point, I assume there are sim­i­lar uncer­tain­ties at play in every large inter­net sys­tem, and I assume that the peo­ple who build and main­tain those sys­tems don’t totally under­stand them. In that way, I think they oper­ate as much like cus­to­di­ans and care­tak­ers as design­ers and engineers.

(It’s worth not­ing that this effect is even more pro­nounced when it comes to AI models, which are sort of nat­u­rally mysterious; for this reason, “AI explainability” is a knotty and impor­tant sub-field.)

Anyway, it feels to me like the ghost fave bug must have something to do with two or more of the many com­po­nents in Twitter’s own mystery machine not talk­ing to each other correctly; the API endpoint duti­fully sig­nals the database, but, elsewhere, a cache isn’t reset … some­thing along those lines. But that’s just a guess. Honestly, I’d love to know if, inside Twit­ter HQ, the bug is totally understood, just not a priority … or if there’s some mystery to it. I’m root­ing for mys­tery.

July 2021, Oakland