Police in Hiroshima prefecture (state) are working with U.S. military officials to investigate the alleged incident, which the woman reported to police, top government spokesman Nobutaka Machimura told reporters.
"It would be unforgiveable if this turns out to be true," Machimura said.
Hiroshima prefectural police spokesman Chikara Nakata confirmed that they were looking into "the incident that has been reported in the media," but refused to discuss any other details.
The woman allegedly met the men, said to be Marines from the nearby Iwakuni Marine Corps Air Station, at a restaurant or bar in Hiroshima early Saturday morning, local media reports said, citing unnamed local police officials.
The men allegedly took the woman outside to a car and drove her to a parking lot around 2 kilometers (1.2 miles) away where they raped her, according to the reports.
The U.S. Embassy said in a statement that it was "aware of the reported incident" and, together with U.S. Forces Japan, was cooperating fully with Japanese authorities. "We take reports of this nature very seriously," it added.
USFJ spokesman Master Sergeant Terence Peck made identical remarks.
Neither officials nor the media reports named any of the people involved in the alleged incident.
The U.S. has about 50,000 troops based in Japan under a bilateral security treaty. Many Japanese complain of crime, pollution and noise associated with the bases.
The rape of a schoolgirl in Okinawa, which has the largest U.S. military presence, by three American servicemen sparked large-scale protests in 1995.
Hiroshima is 687 kilometers (429 miles) southwest of Tokyo.