One special significance to the last three lines
Notice Neptune, though,
Taming a sea-horse, thought a rarity,
Which Claus of Innsbruck cast in bronze for me.
is that the narrator's cold, calculating, and vain nature is clearly revealed. He has turned from speaking with a subdued passion of bitterness:
... Sir, 't was not
Her husband's presence only, called that spot
Of joy into the Duchess' cheek / ... /
... She had
A heart - how shall I say? - too soon made glad,
to speaking of his future bride and her dowry ("no just pretence / Of mine for dowry will be disallowed") to speaking of a statue of Neptune that some artist worthy of name-dropping made especially for him. This clearly shows that, while logical and rational, he has not one grain of human feeling in his cold controlling heart; it is as cold as Neptune and the sea, as controlling as the taming of the magnificent sea-horse.