In this poem, the speaker explains to his lover that he must leave, though not because he has grown weary of her or believes that he can find a better lover out there in the world somewhere. He just has to go on some kind of trip, but he explains, however, that since he "Must die at last," it probably is not a bad idea to get used to the idea of parting from one another. If such short-lived partings are hard, imagine how difficult it will be when they are parted by death.
The speaker goes on to say that man's power is "feeble" because
[...] if good fortune fall, [he]
Cannot add another hour,
Nor a lost hour recall!
In other words, even when a man experiences really good fortune and luck, he can do nothing to prolong that good fortune nor to recall any good times that have already passed. We can do nothing to gain more time or luck for ourselves, and, thus, our power is "feeble" as a result.