The court ruled Russia failed to properly investigate the execution-style killings in a Chechen village that the relative - a daughter of two of the victims - said were carried out by a group of masked men belonging to the Russian special forces.
The ruling - one of a series of recent decisions by the court against Russia in cases concerning Chechnya - found the Russian state violated Europe's human rights convention on four other counts.
The assailants killed Zura Sharaniyevna Bitiyeva, a political figure in Chechnya who participated in anti-war protests, along with her husband, one of their sons and her brother. The four were all shot in the head in the middle of the night by Russian-speaking men.
Bitiyeva's daughter and another of her sons hid from the assailants behind an armchair, the court said.
The court said witness statements indicated the killings were carried out by state servicemen, based on descriptions of the way the killers were dressed, the vehicles they used and the fact that they were able to travel unhindered during curfew hours.
The court also said witness testimony on the assailants methods of working - such as checking passports, putting hoods over detainees' heads and the execution style of the killings - also led it to conclude that Russian personnel were behind the slayings.
Russia has three months to appeal. Dozens of other similar cases are pending before the Strasbourg court.
Chechnya has been torn by two wars pitting Russian forces and their local allies against the rebels. A Moscow-backed government is in power and large-scale battles are now infrequent, but fighting persists.
An estimated 100,000 civilians, soldiers and insurgents have died in Chechnya since 1994. Human rights groups have also reported mass disappearances, blaming them on pro-Moscow Chechen security forces and Russian troops.