The destroyer Kirisame and its crew of about 200 returned to Sasebo port in southwestern Japan to an honor band and a crowd of welcoming family members.
"We have finished our mission, and we are home," said Yuji Fukuhara, the Kirisame's captain.
The refueler Tokiwa will arrive Friday at a Tokyo port, where the Defense Ministry will hold a larger ceremony to be attended by Defense Minister Shigeru Ishiba.
The return of the Kirisame marked the end of a mission that began in 2001 and underscored a deep rift in Japan's government over how far this country should go in supporting the global war on terror.
The mission was abruptly halted on Nov. 1, after opposition parties raised concerns it was too broad and possibly violated Japan's pacifist constitution. Prime Minister Yasuo Fukuda, a staunch supporter of bolstering Japan's security efforts abroad, has vowed to resume the mission quickly.
The U.S. has also been pushing for a resumption, but says the withdrawal of the ships will have no significant impact on operations in Afghanistan.
"Unfortunately, we have to come to a point where we have to welcome them home, but they were engaged in work for a long time in a difficult environment," Japan's Chief Cabinet Secretary Nobutaka Machimura said.
No restart to the mission is expected soon.
Last week, Japan's lower house of parliament approved a resumption of a watered-down mission in the Indian Ocean, but the legislation is now in the opposition-controlled upper house.
The new mission would be limited to refueling and supplying water to craft used in monitoring and inspecting vessels suspected of links to terrorism or arms smuggling. Ships would not refuel coalition vessels directly involved in troop activities in Afghanistan.
During its six-year mission, Japan provided about 490,000 kiloliters (129 million gallons) of fuel in the Indian Ocean to coalition warships, including those from the U.S., Britain and Pakistan, according to the Defense Ministry.
The mission was in line with a push by Japan 's leadership - which is trying to win a permanent seat on the United Nations Security Council - to develop a higher profile in world diplomacy by participating in an increasing number of global peacekeeping missions.
In 2003, Japan sent troops to Iraq for a non-combat, humanitarian mission and sent troops to help in recovery efforts after the devastating tsunami that hit South Asia the following year.
But the push to send troops abroad has been highly controversial. One of the top priorities of the leading opposition party after it won control of the upper house in July elections was to scuttle the naval mission, and its top officials have not shown any signs of backing down.
Polls indicate that the public is gradually coming to accept the idea of overseas operations.
"I think the Japanese people's mindset has changed greatly," said Akira Kato, a defense specialist at Obirin University in Tokyo . "During the Gulf War, sending the troops overseas was considered outrageous. But now people think it is important to strengthen global cooperation through such missions as disaster relief."
He noted, however, that sending troops into actual combat remains taboo.