The keyboardist died early Tuesday, a spokeswoman for Vienna's Wilhelmina Clinic said, without giving details. Zawinul had been hospitalized since last month.
Zawinul suffered from a rare form of skin cancer, the Austria Press Agency cited Risa Zincke, Zawinul's manager, as saying.
Zawinul, who turned 75 on July 7, won widespread acclaim for his keyboard work on chart-topping Davis albums such as "In A Silent Way" and "Bitches Brew," and was a leading force behind the so-called "Electric Jazz" movement.
In 1970, Zawinul founded the band Weather Report and produced a series of albums including "Heavy Weather," "Black Market" and "I Sing the Body Electric." After that band's breakup, he founded the Zawinul Syndicate in 1987.
Zawinul is credited with bringing the electric piano and synthesizer into the jazz mainstream.
This past spring, he toured Europe to mark the 20th anniversary of the Zawinul Syndicate. He sought medical attention when the tour ended, the Viennese Hospital Association said in a statement last month.
Austrian president Heinz Fischer said Zawinul's death meant the loss of a "music ambassador" who was known and cherished around the world.
"As a person and through his music, Joe Zawinul will remain unforgettable for us all," Fischer said in a statement.
Austrian Chancellor Alfred Gusenbauer praised Zawinul's "unpretentious way of dealing with listeners" and said he wasn't "blinded by appearances."
"Wherever he performed, he impressed with his playing," Gusenbauer said in a statement.
Zawinul's son, Erich, said his father would not be forgotten.
"He lives on," Erich Zawinul was quoted as saying by the APA.
Zawinul, who was born in the Austrian capital, emigrated to the United States in 1959, where he played with Maynard Ferguson and Dinah Washington before joining alto saxophonist great Cannonball Adderley in 1961 for nine years, according to a biography on his Web site. With Adderley, Zawinul wrote several important songs, primarily the slow and funky hit "Mercy, Mercy, Mercy."
Zawinul then moved on to a brief collaboration with Miles Davis, just at the time Miles was moving into the electric arena. It was Zawinul's tune "In a Silent Way" that served as the title track of Miles' first electric foray, according to the biography.