Throughout the Duke's monologue he reveals his vile character. He is an egomaniac. He is a name-dropper. He mentions the name Fra Pandolf early in his monologue because this artist was evidently highly regarded and much sought after. He mentions Klaus of Innsbruck partly for the same reason. Note that he ends the line with the words "for me!" with an exclamation mark. But he has another reason for trying to call his visitor's attention to the bronze statue of Neptune taming a sea-horse. His visitor, who was representing the father of the girl the Duke wants to marry, apparently became disgusted with the Duke and jumped up from his chair to leave. We know this because the Duke asks:

Will't please you rise? We'll meet
The company below, then.

The company downstairs probably includes the girl's father and mother and perhaps the girl herself. It would appear that the Duke's visitor became appalled when he realized that the Duke had had his first duchess killed.

I gave commands:
Then all smiles stopped together.

The first duchess's offense was that she smiled too much. She was too happy. We can visualize the Count's representative heading out of the room and toward the stairs with the intention of warning the Count against letting his daughter marry this monster. The Duke calls after him:

Nay, we'll go 
Together down, sir.

Nothing has been accomplished. He had invited this man upstairs on the pretext of showing him part of his art collection, but he really wanted to settle the question of the dowry. He doesn't realize how he has talked himself out of both the girl and the dowry in his monologue. He tries to stall his visitor. He hopes to hold him there a while longer and then bring up the dowry again. He is a vulgar man, as he reveals in many ways, including the clumsy rhymes in the open couplets of the poem--e.g., wall, call ... durst, first ...etc. He values his art collection because the pieces show what a great man he is. He has no real appreciation of beauty. He values the portrait of his dead wife more than he ever valued the lovely girl herself.

His visitor is being intentionally rude to him. No doubt the Count's representative feels it is unwise to risk offending such an important man as the Duke, but he feels such a loathing for wicked man that he can't stay in the same room with him for another minute. After the reader has read this monologue he will be hoping the visitor will persuade the Count not to let his daughter become another possession and probable future victim of this hateful, ignoble nobleman.