The question, which seems simple--though hard to answer--on the surface, really reflects a radical change in historical precepts of the outcome of war. To the victor go the spoils has been debunked and denounced. What is left now is to find new ideas for a creative solution to a new problem that must value the safety of the object as thoroughly as it values the rights of the nation of origin.
When one is dealing with objects of great historical significance as the Elgin marbles, the normal rules don't apply. Although the British purchased them fair and square, they did so from a government that was itself not Greek and had no respect or regard for the heritage of the country. There are often stories of family heirlooms, etc. from Jewish families lost during the holocaust that are now being returned. At times, one must respond to a "higher law" and do that which is right as opposed to that which might otherwise be "legal." For that reason, I sincerely believe that the Elgin marbles should be returned.
The argument for items which belong to the cultural heritage of other nations not being given back as they are safer in the British Museum may be true to an extent but is certainly a very weak one.
I guess I wouldn't be wrong in equating that to a pickpocket defending himself in court with the argument that the rightful owner often mislaid his wallet. To be allowed to make such a claim perhaps the British can approach the UN and we could have a separate court set up that would decide on whether the risks are too high; and the original owners do not have the ability to create infrastructure for the safekeeping of what is returned to them.
Clearly the Elgin marbles are an important symbol of British colonial power and authority. They do not "belong" to Britain in terms of representing British cultural history, and therefore this creates a strong moral argument for their return. The only hesitation I would have would be if this creates a sudden chorus of nations arguing that they should have their items returned as well. In addition, I would suggest that the Elgin marbles, as misplaced as they are, are perhaps more secure and safe in Britain than they would be in Greece.
Morally speaking, the Elgin Marbles should be returned. The only real reason to refuse to return them is if they are not in condition to be moved.
It is true that Elgin acquired the Marbles from what was then the legal government of Greece. However, it should be noted that that government was the government of the Ottoman Empire, not a native Greek government. Therefore, Elgin essentially got an important part of Greece's cultural patrimony from the Turks. Thus, the British Museum cannot really argue that the Greeks voluntarily gave up the Marbles.
The Marbles are clearly part of Greece's cultural history. It is fair that they should be held in Greece so long as Greece has not voluntarily parted with them. Since Elgin's acquisition of the Marbles was not really done with the consent of the Greeks, the Marbles (morally speaking) still belong to Greece and should be returned if it is physically possible.