Enright had been considered a long-shot to take Britain's most prestigious, and contentious, literary trophy. The prize, which carries a check for 50,000 pounds (US$100,000; 75,000 EUR), was awarded during a ceremony at London's medieval Guildhall.
She is the second Irish writer to win the prize in the past three years, after John Banville's "The Sea" in 2005.
"The Gathering" is a family epic set in England and Ireland, in which a brother's suicide prompts 39-year-old Veronica Hegarty to probe her family's troubled, tangled history.
Enright said people looking for a cheery read should not pick up her book.
"It is the intellectual equivalent of a Hollywood weepie," she said.
Howard Davies, the chairman of the judging panel, acknowledged the book was "a little bleak" in places, but praised it as "a very readable novel."
"Anne Enright has written a powerful, uncomfortable and, at times, angry book. 'The Gathering' is an unflinching look at a grieving family in tough and striking language," he said.
Enright said the book's focus on family was a classically Irish theme.
"I think family is a hugely interesting place, it's a place where stories happen. ... And it's also a central Irish institution," said Enright, who admitted feeling a bit "trembly" when she heard she had won.
Jonathan Ruppin of British bookstore Foyles called the judges' choice "a welcome boost for serious literature."
"Not everyone will be comfortable with this bleak account of conflict and despair, but the writing is undeniably exquisite," he said.
Dublin-born Enright, 45, has published three previous novels as well as short stories and the nonfiction book "Making Babies: Stumbling into Motherhood."
Enright's victory was another surprise from an award renowned for unpredictable results. Bookmakers had made Enright the rank outsider to win, with bookies William Hill giving her 20-1 odds.
"Mister Pip" by New Zealand's Lloyd Jones and British novelist Ian McEwan's "On Chesil Beach" had been strongly favored.
But the Booker judges have a history of defying the odds when awarding the prize, which usually brings a huge sales boost for the winner. The favorite has not won since 2002, when Yann Martel's "Life of Pi" took the prize.
Another popular choice that lost out was Mohsin Hamid's "The Reluctant Fundamentalist," the story of a middle-class Pakistani in New York whose relationship with his adopted home changes radically after the Sept. 11, 2001, attacks.
The other nominees were English writer Nicola Barker's "Darkmans," a sprawling supernatural saga set in the town of Ashford in southern England, hailed by the panel as "an ambitious and energetic contemporary ghost story"; "and "Animal's People," a novel about the Bhopal chemical disaster by India's Indra Sinha.
The prize, which is open to writers from Britain, Ireland and the Commonwealth of former British colonies, was founded in 1969 and was long known as the Booker Prize. It was renamed when the financial services conglomerate Man Group PLC began sponsoring it five years ago.
The choice of the city of Helsinki is not incidental as the capital of Finland had hosted US-Soviet negotiations on the limitation of nuclear stockpiles in 1969