“And if you move here, you can live across the street from the guy who wrote The Silence of the Lambs!“

A beat.

“You do want the fabulous writer’s life in the Hamptons, don’t you?”

You know you’re making the right decision to move from the city to the Hamptons when your realtor, on your first journey out to find the perfect bungalow near the beach, spends as much time giving you a historical tour of the crash pads of every writer, dead and alive, who called the Hamptons home as she does finding you your dream cottage. And knows instinctively that living across the street from Thomas Harris, who did in fact create Hannibal Lecter right here in Sag Harbor, isn’t spooky at all.

Keep going, I tell her.

“OK, so this is where John Steinbeck lived… E.L. Doctorow still lives right here. It’s a pretty modest house, considering, but it is E.L. Doctorow.” Joseph Heller. Peter Matthiessen, who worked for three years as a commercial fisherman before becoming the Peter Matthiessen, still lives over there, she says. George Plimpton’s old house, Betty Friedan’s. “You wanna see where Jay McInerney lives?”

Be still, our hearts.

Truman Capote lived and wrote in Sagaponack for 23 years

We had started in Sag Harbor, of course, which is mecca for famous and fabulous Hamptons writers, going back many, many decades. But the entire East End (of Long Island), the nonpretentious, writer-preferred term for the Hamptons, is rife with the spirits, past and present, of literary luminaries: from Truman Capote, who retired to his rustic Sagaponack retreat, with its screened-in porch and single twin bed on four sumptuous acres, to complete In Cold Blood when he wasn’t at the local haunt Bobby Van’s with his buds Matthiessen and Heller, imbibing his favorite drink, an “orange thingee” (four parts vodka, one part orange juice), which often led to Matthiessen having to deliver him back to Sagaponack, to the Pulitzer Prize–winning biographer Robert Caro (his most recent effort: the stellar four-volume work on Lyndon Johnson), who, according to the very local Dan’s Papers, is known to take a thermos and a sandwich out into the toolshed in the woods by his East Hampton home, where he writes in longhand, then types it up later on an old Smith Corona. Writer-wise, it doesn’t get much better than this.

Peter Matthiessen, novelist and naturalist, at home in Sagaponack

The first thing I did when I decided to move from the city to the East End was ring up my two favorite writers-in-residence, authors Michael Shnayerson (currently at work on a bio of Andrew Cuomo for Grand Central Press) and Robert Sam Anson, whom I sometimes refer to as the most fascinating man alive, both for his reportorial skills and his Zelig-like past. (How many people do you know who were not only at the Ambassador Hotel in Los Angeles the night Robert Kennedy was assassinated, but also survived being taken prisoner in Cambodia while reporting for Time?) Anson lives in the woods and on the bay, which is possible to do in Sag Harbor, in a home made more awesome by his wife, the interior designer Amanda Kyser, but he spends most of his time in the backyard in a silver Airstream trailer, a.k.a. his office. You just can’t do this sort of thing in Manhattan. And in a very Hamptons-esque aside, it was Anson who introduced me to Shnayerson—on a blind date, which didn’t take, but enough about that. Bob Sam and Michael would become my Hamptons posse, soon joined by the inimitable Steven Gaines, who single-handedly peeled the layers off the Hamptons and all its marvelous melodrama with his classic, Philistines at the Hedgerow. Which to this day, 15 years after publication, is still prominently displayed and sought after at BookHampton, the independent bookstore to the cognoscenti.

Gaines, who has written 12 books while ensconced in various homes in the Hamptons, remembers his epiphany in 1971. At the time, he was quite the party boy in Manhattan (as evidenced by his juicy roman à clef The Club, about the debauchery at Studio 54), but was trying to write his first book. “Friends with a house on the East End gave me the keys to their house and said, ‘You’re too much of a party boy, you’ll be out all night, you’ll never get serious about writing. Go write the book.’ ” At the time, “there were tumbleweeds all over. And nothing open. You could starve! The train only went to Westhampton [30 miles away], the supermarket closed at 6.” But could you write? “You had no choice.”

Betty Friedan speaks at a party in support of women’s rights in East Hampton, 1970

It is little mystery why the literati, and wannabe literati, head east from Manhattan. For starters, what are the options? New Jersey? The godforsaken New York suburbs? Seriously?! The Hamptons offer that elusive combination of ethereal beauty and intellectual stimulation. If you choose it. And if you don’t, you can hide in your bungalow or your castle and no one will give a rat’s ass. It’s peace, at its finest.

It’s also inspiring. The actress Christina Haag, who wrote the gorgeous best-selling memoir Come to the Edge, about her relationship with John Kennedy Jr., decamped to the East End to write it. “Whenever I get to the dunes and smell the privet and the salt air, I’m anesthetized,” she says. “It was as if it were calling me to write there.” She wrote in Montauk in the dead of winter, then in Sag, then finished the book “in a little dream cottage tucked behind Estia’s,” a favorite restaurant “with a view of Little Long Pond, where I’d hear the loons at night. And I would stumble over to Estia’s for the best coffee out there.” As lonely as writing can be, being here was soothing, says Haag: “I knew this was the place where I could have support, peace, seclusion and anonymity if I wanted it.”

The support writers find out here is an enormous part of the appeal. So integral to the fabric of the Hamptons are the literary heroes who live among the ranks that you’ll find things such as The Grapes of Roth, a wine label by the dean of the local winemakers, Roman Roth—in honor of Steinbeck, of course, whose classic The Winter of Our Discontent is believed to have been inspired by the Schiavoni supermarket family in Sag Harbor.

George Plimpton at the famous Artists-Writers Softball Game in East Hampton, 1989

There is an annual Artists-Writers Softball Game in East Hampton that is such an institution that summer-crowd celebs like Alec Baldwin and Bill Clinton are known to stop by—and even join the game. There is a yearly writers’ gala that is one of the hottest tickets at the beach. And at the Bridgehampton library, “Fridays at 5,” a lecture series in the garden with mostly local authors, is actually competitive, both for the authors themselves and the ticket-seekers. And of course there are the haunts, perhaps none more famous than the American Hotel in Sag, where book editor Alice Mayhew has her own table and McInerney can be found with fourth wife Anne Hearst. It’s where all the writers hang out, even if they’re just visiting, such as the time Fred Exley came to see his editor, Robert Loomis, and ended up being driven home nightly (he stayed a while) by the bartender who could quote passages of A Fan’s Notes.

But it’s the indie bookstores, particularly BookHampton, that really nourish the community—prominently showcasing (and talking up) local writers, few of whom are strangers to the stores. What happens when, say, a James Salter walks in?

Joseph Heller in East Hampton, 1990

“First of all,” says Kim Lombardini, a manager at BookHampton in East Hampton, “you’ve got your authors that you recognize by face, and then you’ve got your authors where someone has to elbow you and say, ‘That’s James Salter.’ ” But do all of them check how prominently their book is displayed? “Oh, yeah, all of them do that!” says Lombardini. One way or another. “I heard from Kurt Vonnegut’s widow, Jill, that he used to send her in to find out where his books were placed in the store while he sat out in the car smoking cigarettes. Mrs. Doctorow has been in. I talked to her and assured her that we had everything in from Mr. Doctorow.”

Among the local favorites on the shelves: Colson Whitehead (a regular at the store), Andy Cohen, Adam Ross, Kati Marton, Ali Wentworth, Alexandra Styron, Heller, Capote, Nora Ephron. The current best-selling local authors? “It’s between Nelson DeMille and Alan Furst right now—and Ina [Garten]’s cookbooks.”

I ended up not moving next door to The Silence of the Lambs but into the most adorable cottage in Bridgehampton that’s a stone’s throw from Bobby Van’s, where my not-so-shabby neighbors include James Salter. I knew one of my favorite editors had once lived in my cottage but recently learned the entire parcel of land it sits on was once owned by the novelist Barbara Howar, during her tumultuous three-year marriage to Willie Morris. Which is the other great thing, of course—there’s a story behind every privet.