The announcement Tuesday that O'Neal is retiring immediately came days after the world's largest brokerage posted a $2.24 billion (EUR1.55 billion) quarterly loss, its biggest since being founded 93 years ago.
Merrill Lynch & Co. did not name a replacement for O'Neal, whose ouster had been expected for days. Instead, board member Alberto Cribiore will step in as interim nonexecutive chairman and head of the search committee to find a new CEO.
Despite the third-quarter loss, analysts consider the vast majority of Merrill's business to be in perfectly fine shape. But, whoever replaces O'Neal will have to clean up the segment that is not in good order - Merrill's investments in subprime mortgages and other risky types of debt.
O'Neal, 56, who rose to power five years ago, was known for shaking up top management and putting a greater emphasis on riskier bets, rather than the safety of just selling stocks.
That strategy - which handed Merrill Lynch record results during the market's peak - came with a heavy cost during the tumultuous third quarter.
O'Neal ultimately shouldered the blame for the earnings miss. With his retirement, he becomes the biggest executive casualty of the credit crisis that swept global markets this summer.
"Mr. O'Neal and the board of directors both agreed that a change in leadership would best enable Merrill Lynch to move forward and focus on maintaining the strong operating performance of its businesses, which the company last week reported were performing well, apart from sub-prime mortgages and CDOs," Merrill Lynch said in a statement.
CDOs, or collateralized debt obligations, are complex instruments that combine slices of different kinds of risk. It was Merrill's bet on CDOs, and the subprime mortgages underpinning many of them, that proved to be O'Neal's downfall.
Analysts have said this week that whoever replaces O'Neal may have to write down another $4 billion (EUR2.78 billion) worth of bad investments in the fourth quarter.
Merrill Lynch said that co-presidents and chief operating officers Ahmass Fakahany and Gregory Fleming will continue in those positions. Fleming will lead Merrill's front office, such as its brokerage and investment banking sides. Fakahany, a close confidant of O'Neal, will lead back-office functions such as finance and human resources.
It was not known how much O'Neal would receive as an exit package, though there have been some reports it would be nearly $200 million (EUR139 million). He was paid roughly $48 million salary in 2006, and had $160 million in stock and retirement benefits, according to James Reda, founder of compensation consultancy James F. Reda & Associates.
O'Neal is the descendant of a former slave, and grew up in poverty in Alabama before rising to become one of the highest-ranking African-Americans on Wall Street. He worked his way through a Harvard business degree by working at General Motors Corp., and in 1986 joined Merrill as a banker in its junk-bond department.
His elevation to CEO was seen by some as an experiment by the company's board, most of which have since retired. O'Neal mostly held positions on the client-contact side, which goes against the trading background most of its other CEOs had.
Among those said to be considered for the job outside the firm are NYSE Euronext CEO John Thain and BlackRock Inc. CEO Laurence Fink. Fleming is also considered a possible internal candidate.
Merrill Lynch shares fell $1.06 to $66.36 in morning trading.