There is only one direct, if somewhat ambiguous, reference to masculinity in William Shakespeare's Hamlet.
When the audience first sees Hamlet, he's dressed in black mourning clothes, walking around with his head down and looking grieved because of his father's recent death.
Hamlet's mother, Queen Gertrude, and his uncle, King Claudius, now his stepfather, both remark on his appearance. They seem solicitous of Hamlet, but they're clearly frustrated and annoyed by his mood and his choice of clothing:
CLAUDIUS. How is it that the clouds still hang on you?...
GERTRUDE. Good Hamlet, cast thy nighted color off,
And let thine eye look like a friend on Denmark.
Do not for ever with thy vailed lids
Seek for thy noble father in the dust. (1.2.68–73)
Claudius says that he understands Hamlet's feelings about his father's death, but he's grown impatient with Hamlet's incessant show of grief, whether it's "actions that a man might play," as Hamlet says, or truly heart-felt:
CLAUDIUS. But to persever
In obstinate condolement is a course
Of impious stubbornness; 'tis unmanly grief;
It shows a will most incorrect to heaven,
A heart unfortified, a mind impatient,
An understanding simple and unschool'd...
Fie! 'tis a fault to heaven,
A fault against the dead, a fault to nature,(105, emphasis added)
To reason most absurd... (1.3.95–106)
It's unclear by his use of the words "unmanly grief" if Claudius is simply making a reference to accepted male behavior, or if he's actually insulting Hamlet's masculinity.
Either way, Hamlet takes no notice of the remark, and Claudius never mentions Hamlet's manliness, or lack thereof, again.
In his soliloquy directly following the scene with his mother and uncle, Hamlet seems entirely comfortable mocking his own seeming lack of manliness:
HAMLET. Like Niobe, all tears—why she, even she—
O God! a beast that wants discourse of reason
Would have mourn'd longer—married with my uncle,
My father's brother, but no more like my father
Than I to Hercules. (1.2.152–156)
The words "masculine" and "feminine" don't appear in the play. All references in the play to "man," whether to a particular man or to men in general, are descriptive, not judgmental.
However, Hamlet does disparage women, in general and in particular:
HAMLET. ...Frailty, thy name is woman! (1.2.149)
OPHELIA. 'Tis brief, my lord.
HAMLET. As woman's love. (3.2.142–143)
HAMLET. It is but foolery; but it is such a kind of gain-giving
as would perhaps trouble a woman. (5.2.207–208)
In saying these things against women, however, Hamlet doesn't say them to assuage his ego, or to prop up his own masculinity.
Laertes makes one remark about women when he hears that his sister, Ophelia, has drowned, and he's overcome with grief:
LAERTES. Too much of water hast thou, poor Ophelia,
And therefore I forbid my tears; but yet
It is our trick; nature her custom holds,
Let shame say what it will. When these are gone,
The woman will be out. (4.7.201–205)
This is fairly mild, however, and relates more to Laertes's own release of emotion—which he fully accepts as part of his own nature—which he's holding back until he can take revenge against Hamlet for his Polonius's death.
It would appear that Shakespeare doesn't address the issue of masculinity in Hamlet, but lets the audience judge each character's behavior for themselves, based on each audience member's individual standards of masculinity and femininity.