Because this poem centers on the contrast between arrogance, jealousy, and control, on one hand, and purity, joy, modesty and kindness, on the other, one can argue that the portait of the Duchess and the sculpture of Neptune taming the sea-horse are both important symbols.
Either through death by nature or misadventure (that is, murder), the Duchess is gone from the Duke's world, and what is left is her portrait, which has been so skilfully painted that a viewer can perceive her nature, and therefore the portrait itself becomes a symbol of not only the Duchess herself but a blighted life. Through the Duke's negative commentary on the Duchess, we can conclude, among other things, that she loved life; she was generous; she took pleasure in many things; she extended kindness to everyone, irrespective of rank. The portrait remains as a symbol of all those positive, life-affirming traits that contrast so starkly with the Duke's negative and destructive behaviors.
The delicate blush, captured so well by the painter, Fra Pandolf, is a symbol within the larger symbol of the portrait because it is emblematic of the Duchess's innocence and generosity, and it remains as the only evidence of the Duchess's good nature.
The sculpture of Neptune taming the sea-horse is also a perfect symbol of the Duke's arrogant and controlling nature. The sea-horse, like the Duchess, is a symbol of innocence; Neptune, the Roman god of the sea, symbolizes the controlling and harsh nature of the Duke. Of course, the Duke, who is blind to his own nature, would never perceive the irony inherent in that particular sculpture. If anything, the statue, which, he makes sure his visitor understands, was made just for him, symbolizes to the Duke the natural order of things--control of everything.