The decision will not block plans for the 1,200-kilometer - or 750-mile - pipeline, which will deliver natural gas from Russia to Germany under the Baltic Sea.
It will, however, prevent the consortium, Nord Stream, from considering a route that passes a few hundred meters into Estonian territorial waters. Experts say that section of seabed might provide a more environmentally safe route for that part of the pipeline.
"We think the Baltic Sea is not the proper place for such a pipeline," Foreign Minister Urmas Paet said at a news conference. "Each coastal country has full sovereignty and a right to make decisions involving its own waters."
In explaining the decision, Paet cited environmental concerns and possible hazards in the Baltic Sea, which in recent years has become increasingly polluted and seen tanker traffic surge.
"It would be better if the pipeline were located on land instead of under the sea," he said.
Paet said the survey would have required excessive drilling, which is forbidden by Estonian law. Also, the survey could have revealed sensitive information about the size of Estonia's natural resources, the minister said.
The EUR5 billion (US$7 billion) project has a 2010 launch date. The current route proposal, approved last year by the consortium, would not take the pipeline into Estonia's territorial waters and would not be affected by Thursday's decision.
Switzerland-based Nord Stream AG said its request to carry out a geological survey in Estonian waters was in response to a Finnish request for the company to explore ways to minimize the project's potential environmental impact, said company spokesman Jens Muller.
Finnish experts believe that part of the seabed could be flatter and thus more environmentally sound for the pipeline, Muller explained. "We're talking about a distance of several hundred meters" into Estonian territory, he said.
Estonia and other Baltic Sea nations say Nord Stream has not consulted closely with them on the project, leading to some annoyance. Nord Stream contacted Estonia about the project for the first time only in May, Paet said.
Several other coastal nations have also expressed concern that the pipeline could damage the Baltic Sea's delicate ecosystem.
Among Estonians, the project has been highly unpopular, according to recent surveys.
Relations between Estonia and Russia took a hit this year after the Baltic state removed a Soviet war memorial - which held deep meaning for ethnic Russians - from downtown Tallinn.
According to Nord Stream's plans, after completion of the project's first phase in 2010, the pipeline will deliver 27 billion cubic meters of gas to Germany. Two years later a parallel pipeline will double capacity to 55 billion cubic meters.
On the second day of the St. Petersburg International Economic Forum, a plenary meeting was held, in which Russian President Vladimir Putin, French President Emmanuel Macron, Japanese Prime Minister Shinzo Abe, Chinese Vice President Wang Qishan and IMF head Christine Lagarde took part