Writing with the machine

I made some­thing new: a plu­gin that pro­vides inline text com­ple­tions pow­ered by an AI language model.

rnn-writer example

Building this felt like play­ing with Lego, except instead of plas­tic bricks, I was snap­ping together conveniently-packaged blocks of human intel­lect and effort.

One block: a recur­rent neural network, fruit of the deep learn­ing boom, able to model and gen­er­ate sequences of char­ac­ters with spooky verisimilitude. Snap!

Another block: a pow­er­fully exten­si­ble text edi­tor. Snap!

Together: responsive, inline “autocomplete” pow­ered by an RNN trained on a cor­pus of old sci-fi stories.

If I had to offer an extrav­a­gant anal­ogy (and I do) I’d say it’s like writ­ing with a deranged but very well-read par­rot on your shoulder. Any­time you feel brave enough to ask for a suggestion, you press tab, and … 

rnn-writer example

If you’d like to try it yourself, the code is now avail­able, in two parts:

You’ll find instruc­tions for both tools on their respec­tive Github pages, and if you have dif­fi­cul­ties with either, feel free to open an issue or drop me a line.

Mainly, I wanted to share those links, but as long as I’m here I’ll add a few more things: first a note on motivations, then an obser­va­tion about the deep learn­ing scene, and finally a link to the sci-fi cor­pus.

The vision

From my first tin­ker­ings with the torch-rnn project, gen­er­at­ing goofy/spooky text mim­icry on the com­mand line, I was struck — almost overwhelmed — by a vision of typ­ing nor­mally in a text edi­tor and then sum­mon­ing the help of the RNN with a keystroke. (When I say “help,” I mean: less Clippy, more séance.)

After fum­bling around for a few weeks and learn­ing five per­cent of two new pro­gram­ming languages, I had the blocks snapped together; the RNN trained; the vision realized. And then my first hour play­ing with it was totally deflating. Huh. Not as cool as I imagined it would be.

This is an unavoid­able emo­tional waysta­tion in any project, and pos­si­bly a cru­cial one.

As I’ve spent more time with rnn-writer, my opin­ion has — er — reinflated somewhat. I am just so com­pelled by the notion of a text edi­tor that pos­sesses a deep, nuanced model of … what? Every­thing ever writ­ten by you? By your favorite authors? Your nemesis? All staff writ­ers at the New Yorker, present and past? Every­one on the internet? It’s provoca­tive any way you slice it.

I should say clearly: I am absolutely 100% not talk­ing about an edi­tor that “writes for you,” what­ever that means. The world doesn’t need any more dead-eyed robo-text.

The ani­mat­ing ideas here are augmentation; partnership; call and response.

The goal is not to make writ­ing “easier”; it’s to make it harder.

The goal is not to make the result­ing text “bet­ter”; it’s to make it different — weirder, with effects maybe not avail­able by other means.

The tools I’m shar­ing here don’t achieve that goal; their effects are not yet suf­fi­cient com­pen­sa­tion for the effort required to use them. But! I think they could get there! And if this project has any con­tri­bu­tion to make beyond weird fun, I think it might be the sim­ple trick of get­ting an RNN off the com­mand line and into a text edi­tor, where its out­put becomes some­thing you can really work with.

Deep scenius

Like any tech-adjacent person, I’d been read­ing about deep learn­ing for a cou­ple of years, but it wasn’t until a long con­ver­sa­tion ear­lier this year with an old friend (who is eye-poppingly excited about these techniques) that I felt moti­vated to dig in myself. And, I have to report: it really is a remarkable com­mu­nity at a remark­able moment. Track­ing papers on Arxiv, projects on Github, and threads on Twitter, you get the sense of a group of peo­ple nearly trip­ping over them­selves to do the next thing — to push the state of the art forward.

That’s all buoyed by a strong (recent?) culture of clear explanation. My excited friend claims this has been as cru­cial to deep learn­ing’s rise as the (more commonly-discussed) avail­abil­ity of fast GPUs and large datasets. Hav­ing ben­e­fited from that cul­ture myself, it seems to me like a rea­son­able argument, and an impor­tant thing to recognize.

Here are a cou­ple of resources I found especially use­ful:

149,326,361 characters

Most of the energy in the deep learn­ing scene is focused on what I’d call “generic” problems, the solu­tions to which are very broadly use­ful to a lot of peo­ple: image recognition, speech recognition, sen­tence translation … you get the idea. Many of these prob­lems have asso­ci­ated bench­mark challenges, and if your model gets a bet­ter score than the reign­ing champ, you know you’ve done some­thing worthwhile. These chal­lenges all depend on stan­dard datasets. And these — datasets — are — extremely boring.

So, a large part of the work (and fun) of apply­ing the deep learn­ing scenesters’ hard-won tech­ni­cal tri­umphs to weird/fun objec­tives is track­ing down non-stan­dard, non-boring datasets. For me, decisions about the col­lec­tion and process­ing of the text cor­pus have been more con­se­quen­tial than deci­sions about the RNN’s design and subsequent training.

The corpus I’ve used most is derived from the Internet Archive’s Pulp Magazine Archive: 150MB of Galaxy and IF Mag­a­zine. It’s very noisy, with tons of OCR errors and plenty of adver­tise­ments mixed in with the sci-fi stories, but wow there is a lot of text, and the RNN seems to thrive on that. I lightly processed and nor­mal­ized it all, and the com­bined corpus — now just a huge text file with­out a sin­gle soli­tary line break — is avail­able on the Inter­net Archive.

So, in conclusion:

Snap. Snap. Snap!

May 2016, Oakland