Marcellus says this. He is one of the guards on duty when the ghost of young Hamlet's father, King Hamlet, appears. Marcellus has seen the ghost of the former King, who died under mysterious circumstances. Marcellus means that there are strange, corrupt and evil events occurring: "something rotten." His remark is aimed at the current King Claudius.
Hamlet has learned from Horatio that the country is preparing itself for war as a reaction against the new King, so, now that he is back from university in Wittenberg, Hamlet sees many things occurring in Denmark that are not good. Prince Hamlet is going to speak to the Ghost, which has beckoned him, despite the warnings from the others not to do so.
Unknown to Hamlet and to the audience is the irony of that statement, "something is rotten in the State of Denmark." Something is definitely rotten, evil and corrupt in Denmark, and that something is King Claudius: Claudius killed his own brother because of his ambition and his desire for that brother's wife.
Since the Ghost is first seen by the watchmen, Shakespeare sets up the entire theme for this play in the quote spoken by Marcellus. He refers to King Claudius when he speaks of rottenness in Denmark, which can mean any kinds of moral evils and political corruptions.
Seeing the Ghost becomes a more developed point further in the play. The Ghost is, obviously, not a figment of Hamlet's imagination but an entity seen by others outside the royal house, for example, by the watchmen.
There is much "rotten in the State of Denmark" including the murder of King Hamlet by his brother, Claudius. Hamlet's mother, Gertrude, has wedded and bedded Claudius when the King is "but two months dead!"
Hamlet, then, is forced to deal with his father's murder and what he believes is his mother's adultery. At the close of the play, the 'rottenness' in Denmark leads to the demise of all the Royals and the end of the royal lineage.
When he speaks of what is rotten in Denmark, he is referring to the corruption found within the land. Gertrude married Claudius and bedded him when her old husband did not die so long ago. Not only that, in a way its hints to the fact that Claudius was the murderer of the King, there are rotten affairs here, political and morally.