The wooden fishing boat went down Sunday near Saint Martin's island, about 120 kilometers (75 miles) south of the coastal resort town of Cox 's Bazar, said local police officer Mohammad Jasimuddin, who is coordinating the rescue effort.
Survivors said the boat was carrying more than 100 people, Jasimuddin said. Only one body has been recovered.
He said more than 50 people were still unaccounted for, adding that about 50 others swam ashore or were rescued by local fishing boats.
Police and coast guards used speed boats to scour the sea for the missing, he said, without providing further details.
Jasimuddin said the passengers were Bangladeshi and Myanmar refugees from camps at Cox's Bazar, 300 kilometers (190 miles) south of the national capital, Dhaka .
Police suspect human traffickers had been using the boat to try taking the passengers to either Thailand or Malaysia, said Banaj Kumar Majumdar, another police official in the area.
Several thousand Myanmar refugees, mostly Muslims known as Rohingyas, have fled to Bangladesh over the years, claiming persecution by Myanmar 's military junta and economic hardships.
In the last three months, police and the coast guard have arrested about 500 people - Bangladeshis and Myanmar refugees - in the same waters, mainly on trafficking or illegal entry charges.
Such accidents are common in Bangladesh , a delta nation with about 250 rivers. They are often blamed on poor navigation, unfit vessels and lax enforcement of safety regulations.