The poem is an elegy to Milton's friend Edward King, who drowned while young. It is also a pastoral elegy, placed in a rural setting. In it, King is envisaged as having been a shepherd living in nature. Much of the poem is spoken by an imagined "uncouth swain," or shepherd friend of King's, lamenting his death.
In this passage, the shepherd is saying that the desire for fame draws men away from the pastoral life of pleasure, such as dallying with beautiful maidens like Amaryllis. Fame, instead wishes:
To scorn delights and live laborious days
The shepherd also says that
Fame is ...That last infirmity of noble mind
By this he means a noble person will put aside all other vices before abandoning the desire to be famous, because it can seem noble to pursue fame. The shepherd, however, questions whether it is worth the price to pursue fame rather than to enjoy the simple life. He goes on to say that lasting fame is not, anyway, bestowed on people in their lifetimes or by other people. It comes after death and is decided on by Zeus ("Jove"_):
As he pronounces lastly on each deed
The shepherd argues, as a shepherd would, that fame is not worth pursing (it is an "infirmity") and is out of our hands.